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FOREWORD 



This report is the result of a special study which was carried out at the 
System Development Corporation offices in Santa Monica, California. The study 
was a portion of the contract work for the BUIC III System Program during the 
period of July 1968 to June 1969. The purpose of the study was to provide a 
documentary record of the contractor's experiences in applying Air Force systems 
management techniques to the BUIC III computer program acquisition. 

Material contained in this report represents a revision of material ori- 
ginally reported in the SDC Technical Memorandum TM-4223, "Systems Management 
Applied to Information Processing Elements in BUIC III; Review of Experience," 
dated 25 February 1969. The content is based on information which was collected, 
organized, and interpreted by the contractor; it does not reflect official 
information provided by the 416M System Program Office (SPO) nor technical 
participation by SPO personnel. 

Administrative monitoring of the task was accomplished by Major Harvey B. 
Blanton and Captain James E. Puffer, Office Symbol ESSGB, of the Air Weapons 
Surveillance and Control System Program Office (416M/P/418L) , Electronic 
Systems Division, Air Force Systems Command, Laurence G. Hanscom Field, 
Bedford, Massachusetts . 

Publication of this report does not constitute Air Force approval of the 
report's findings or conclusions. It is published only for the exchange and 
stimulation of ideas. 
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ABSTRACT 

This report is a review and analysis of experience with the 
application of Air Force systems management techniques to the 
acquisition of information processing elements in the 416M 
(BUIC III) system program. The report includes a background 
review of the systems management concepts and trends in relation 
to practices which had been employed in L-system programs pre- 
ceding BUIC III. Novel requirements introduced in BUIC III 
are identified in the areas of computer program configuration 
management, standard documentation, design reviews, and Category 
I testing; and a summary is presented of the milestones associated 
with these requirements as they actually occurred during the 
program. Finally, the experience in specific areas is discussed 
and evaluated with respect to implications for future modification 
and use of the management techniques. 
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CHAPTER I 
INTRODUCTION 



A. SCOPE AND OBJECTIVES 

This report is concerned with the problem of the integration of a new type of 
technology and the management process in the military realm. The technology 
is that of computer-based command and control systems. The particular focus 
of this report is upon the computer programs and associated products. Manage- 
ment, in this context, refers to the process whereby electronic command and 
control systems are acquired by a procurement agency for a using command. The 
system on which we shall focus is known as BUIC III (416M) , the back-up inter- 
ceptor control system for the SAGE (416L) system of air defense. 

The procurement agency in this example of the system acquisition process is 
the Electronic Systems Division (ESD) of the Air Force Systems Command (AFSC) 
while the Air* Defense Command (ADC) is the user. Two non-military, not-for- 
profit organizations of critical concern in this report are the MITRE Corpora- 
tion, which was the General Systems Engineering/Technical Direction Contractor 
(GSE/TDC) in BUIC III, and the System Development Corporation (SDC), which was 
responsible for the BUIC III operational computer program system segment. 
Other non-military organizations, of course, also played important roles in 
the development of BUIC III. Foremost among these is the Burroughs Corporation, 
which designed and manufactured the system equipment. However, since the focus 
of attention in this report is on the acquisition of computer programs, the 
roles of these other organizations will be dealt with only insofar as they 
impinge upon the main areas of concern. 

In the BUIC III system project, several management techniques based on the 
AFSCM 375-series and associated systems management principles were applied on 
a broad scale for the first time to the acquisition of information processing 
elements of a command and control system. While some of the techniques have 
now been adopted as formal Air Force requirements for general use, BUIC III 
represents a first and significant case of actual experience with their 
systematic application in a major system project. Because of the increasing 
importance of information processing elements in Air Force systems, that 
experience constitutes an invaluable source of potential guidance for future 
applications, as well as for continued expansion, modification, and refinement 
of the techniques. It is therefore a primary objective of this report to 
document the history of the program in such a way as to make the lessons 
learned available for early dissemination and use. 



* Now Aerospace Defense Command. 



B. METHODS 

Two main types of sources were used to gather the information upon which this 
report is based: (1) documents of various types, and (2) interviews with SDC 
personnel who participated in the BUIC III system program. 

The documents used to reconstruct the history of the BUIC III acquisition 
effort and the problems encountered by SDC during the course of that effort 
were derived primarily from SDC sources. These included: historical reports 
on SAGE, BUIC II, and BUIC III written by SDC personnel; letters, notes, and 
memoranda obtained from SDC personnel who participated in a variety of capa- 
cities in the BUIC III effort; similar types of documents obtained from SDC's 
BUIC Management File ; the BUIC III/SDC Configuration Index , SDC's BUIC Monthly 
Management Report going back to July, 1964, SDC's Technical Objective and 
Plan series, SDC's Air Defense Division's annual Activities Report, SDC's 
Lexington Liaison Office Activity Reports and a variety of technical documents 
(SDs, TMs, FNs, Notes). Important sources of information were letters, notes, 
and memoranda prepared by SDC personnel who gathered these documents while 
investigating a particular problem at the request of an SDC manager. 
Additionally, extensive use was made of the files of SDC's System Management 
Project, which has provided continuing support to ESD during the past 5-6 
years in developing the computer program acquisition guidelines, and whose 
members have also conducted special studies associated with the BUIC II and 
BUIC III programs. 

In addition to the document sources, much information in this report has been 
supplied by SDCers who participated in the BUIC III program and provided first- 
hand data on the application of the 375-series to th^ design and development 
of the operational computer program system segment. Material for Chapter VI, 
Discussion and Evaluation, was initially gathered in a series of interviews 
and group meetings with SDC technical and managerial personnel in the Military 
System Division. Their number was too large to permit listing of all of their 
names. These people not only provided essential basic information but also 
contributed useful comments and corrections as the writing progressed through 
several drafts. Helpful factual comments were also supplied by several members 
of MITRE Corporation. 

In working with this material and benefitting from the many comments and sug- 
gestions received, the three authors wish to make it plain that the interpreta- 
tions and viewpoints expressed in the report are their own. 



CHAPTER II 
BACKGROUND OF SYSTEMS MANAGEMENT FOR COMMAND AND CONTROL SYSTEMS 

A. THE PROGRAM DEFINITION PHASE 

On 14 January 1963, the Office of the Secretary of Defense issued a draft 
instruction entitled "Program Definition." The concept of "program definition" 
was based upon the experience of AFSC's Ballistic Systems Division (BSD). The 
formal presentation of the concept appeared in BSD Exhibit 62-101, System 
Analysis: Procedures for System Definition , published in June, 1962, for the 
Mobile Mid-range Ballistic Missile (MMRBM) program. BSD exhibits had pre- 
viously been forerunners of military management procedures with Air Force-wide 
applicability and this proved to be the case in this instance. The BSD exhibit 
proposed a new "system definition" phase. Phase I, to cover the first four 
months of the Acquisition Phase. The Department of Defense instruction 
required such a phase for all large-scale military system development programs, 
not merely those within BSD or the Air Force. 

The new phase was defined as follows in the draft DOD instruction:* 

"Program Definition is the formal process whereby preliminary 
engineering and management planning are accomplished in order 
to arrive at definite performance specifications and refined 
cost and schedule estimates for the project under consideration. 
It is a funded effort by one or more contractors, working in 
close collaboration with the government, having as essential 
objectives the achievement of best technical approaches, the 
identification of high risk areas, and the development of an 
adequate basis for firm fixed price or incentive contracting 
of the main body of a major development project." 

Headquarters USAF responded quickly to the DOD draft instruction and then 
turned the matter over to the Air Force Systems Command for detailed study. 
Headquarters AFSC arranged for a Command-wide meeting to be held at Andrews 
AFB on 6 and 7 February 1963 to review the impact of the new concept on 
existing management policies and procedures. Representatives of the Depart- 
ment of Defense as well as the Army and the Navy also attended and participated. 



The expression "Program Definition" was changed to "Project Definition" and 
subsequently to "Contract Definition" in successive revisions to the original 
draft instruction, which were issued in February 1964 and July 1965 as 
DOD Directive 3200.9. 



It is evident in the report of the Command-wide meeting that the cost concerns 
of the Department of Defense were a primary factor in the publication of the 
draft instruction on Program Definition. It was closely related to the 
experimentation then going on throughout the aerospace industry with a new 
management tool known as Program Evaluation Reporting Technique (PERT) . A 
variation of PERT called PERT/Cost was clearly exciting to DOD personnel since 
it provided "a single system for collating the three things that one has to 
manage in a program. . .the work to be accomplished, the time schedule. . .and the 
cost...."* Mr. Robert S. Tucker, Assistant Director for Engineering Management 
in Department of Defense Research and Engineering (DDR&E) , who presented this 
interpretation of the DOD position, went on to say that PERT/Cost seemed to be 
a powerful tool for managing large development programs of the type represented 
by TITAN III, then under development. At this point it is worth quoting a 
later section of the report on PERT/Cost in full;** 

"About two years ago (1961) when people really began to worry about 
bad cost estimates and the spiralling cost of weapon systems, the 
discussions always centered about the very beginning of the program. 
It was stated that most of our problems were caused because we 
hadn't had a good cost estimate to start with. Everybody seemed to 
agree that this was so. They further agreed that you couldn't get 
a good cost estimate unless you had a good statement of work to 
define the job clearly. We began by thinking along the lines of 
how we could get a better statement of work at the beginning of 
a program and, therefore get a better cost estimate." 

The objective of the Program Definition Phase, then, was to obtain accurate 
cost estimates, and the procedure for achieving this was by associating costs, 
not with some "combination of functions and hardware" but by analyzing the 
system into subsystems and then into specific pieces. These pieces are 
identified as "hardware end-items" and, on the basis of these end-items, PERT 
networks are created. This approach was applied during the TITAN III program 
in the belief that it would provide "a firm basis for negotiation of definitive 
contracts with the contractors when they were selected."*** In this way, the 
contractor's cost estimates would be related to explicitly defined items of 
work instead of an amorphous work statement. Phase I of the Acquisition Phase 
required competition among contractors for the award of development contracts. 
It was believed that PERT/Cost would provide a basis for evaluating competing 
proposals from industry. 



* The Program Definition Phase , Report of a Command-wide Conference held at 
Hq- AFSC, Andrews AFB, 6 and 7 February 1963, p. 6. 
** Ibid., pp. 51-52. 
*** Ibid., p. 7. 



The use of PERT/Cost approach in the TITAN III program was regarded as very 
successful by the Department of Defense and it asked for a similar approach to 
the development of BSD's Mobile Mid-range Ballistic Missile (MMRBM) program 
then just getting underway. In this case, DOD referred to the period of 
system definition as the "Program Definition Phase" and that, according to 
Mr. Tucker, is how the name came into being. 

In essence, the value of the Program Definition Phase, according to the Depart- 
ment of Defense, was that it would make possible the identification of the 
components of a large-scale program before it was undertaken, rather than 
after it was contracted for, and it would provide a means for controlling the 
program, especially costs, after it was begun. 

B. IMPACT OF PROGRAM DEFINITION CONCEPT ON ESD 

Prior to the publication of the DOD draft instruction on Program Definition 
and the AFSC-sponsored Command-wide meeting in February, 1963, the Electronic 
Systems Division had initiated efforts to improve the procurement procedures 
for large-scale, computer-based, command and control systems. In August, 1962, 
personnel representing ESD, MITRE Corporation, and System Development Corpora- 
tion met at Headquarters, ESD, and established a Task Group to examine the 
problems associated with the acquisition of the L-systems which were ESD's 
responsibility. MITRE and SDC were not for profit organizations providing 
ESD with technical support — MITRE for over-all system engineering and SDC 
for computer programs and associated products, a "package" referred to 
at that time by the generic term "software." 

The reasons for the creation of the Task Group may be briefly summarized: (1) 
pressure from the DOD and USAF for tighter and more effective control over the 
procurement process; (2) ESD's concern over the adequacy of its own management 
process; and (3) concern by the ESD SPOs about managing a procurement process 
for "software" aspects of military systems, aspects which were relatively 
unfamiliar to them and which had been managed up to that time by the con- 
tractors. 

The ESD/MITRE/SDC Task Group was composed of a Steering Committee and, over a 
period of time, a number of ad hoc Working Groups. The Steering Committee was 
composed of senior representatives from the three participating organizations. 
Responsibilities of the Committee were to identify problem areas that needed 
investigation, to define specific tasks for the Working Groups, to provide the 
groups with guidance, and to review their work. 

The first Working Group, composed like the Steering Committee of representatives 
from ESD, MITRE, and SDC, and under the chairmanship of then Lt . Col. S. G. 
Morgan of ESD, studied the acquisition histories of 465L, 473L, 425L, 416L, 
and 412L to attempt to identify the typical problems encountered in their 
management and to assess the adequacy of the management techniques used to deal 
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with them. The Working Group reported the results of its investigation in 
October 1962. The following areas were described as major sources of problems 
in need of additional study and improvement: 

Management tools and techniques, i.e. , specifications, contract 
work statements, concurrence documents, etc., for acquiring computer 
programs, associated products, and related technical data. 

User relationships to development agencies and user participation 
throughout the system development process. 

Capabilities of ESD military personnel in the fields of system 
design and computer programming. 

The philosophy and concepts of system design. 

A second Working Group was created to address itself to the problem areas 
identified by its predecessor with a particular focus on the first area, 
management tools and techniques. The task assigned to the Group was to "pre- 
pare drafts or outlines of ESD Exhibits which would serve as guidance and 
illustration in the use or applicability of specific documentation, procedural 
steps, processes and definitions for the uniform conduct of management of com- 
puter program design and acquisition ...." The results of this Working Group's 
efforts were published on 18 February 1963 as a MITRE document, TM-3551, 
Computer Program Acquisition Study. This document constituted a good beginning, 
despite some deficiencies, in what was to become a long-range task of detailed 
definition of the procurement process for computerized systems. It identified 
the major problems associated with the acquisition of computer programs and 
it recommended possible solutions to them. 

Meanwhile, significant events were also taking place elsewhere in Systems 
Command. Following the Command-wide meeting at Headquarters AFSC in February, 
1963, General Schriever established a system Definition Task Force (SDTF) on 
4 March 1963. The "charter" for the SDTF made it clear that the AFSC Commander 
considered the work of the Task Force to be of the utmost importance and to 
merit top priority support from all AFSC divisions. The first meeting of the 
SDTF was held a week later at Headquarters BSD, which had been made the lead 
division for the task at hand. An interdlvisional team was formed as a result 
of this meeting and it began full time work at Norton AFB, California, on 
1 April 1963. Lt. Col. S. G. Morgan was the ESD representative. 

The immediate objective of the Task Force was to incorporate a Definition Phase 
into the typical system life cycle, in compliance with General Schriever 's 
directive to implement the DOD instruction on that concept, and to revise 
existing Air Force systems management documentation accordingly. However, 
after studying the problem, the SDTF undertook the reorganization and 



consolidation of the entire systems management framework. This task took the 
form of initiating the development of a guidance manual for general use by the 
System Program Offices throughout AFSC, which would contain uniform procedures 
covering all phases of the system life cycle and all major aspects of systems 
management. This manual (which eventually became AFSCM 375-4) would presumably 
be applicable to electronic command and control systems as well as to jet air- 
craft, missiles, and space vehicles. 

While these events were transpiring at the Command level, the activities of 
the ESD/MITRE/SDC Task Group were interrupted by the receipt of a letter from 
AFSC, dated 20 March 1963, requesting the Division's comments on the DOD draft 
instruction on Program Definition and the BSD Exhibit 62-101 on the same 
subject. Senior ESD, MITRE, and SDC representatives met to consider how to 
reply. 

Various staff groups were asked to review the DOD instruction and BSD exhibit. 
The consensus was that, although there was no question about the desirability 
of a "program definition" phase to precede acquisition, the proposed procedures 
were based upon experience with weapons systems and were not suitable for 
command and control systems. Both MITRE and SDC recommended to ESD that it 
resist any attempt to establish a single type of program definition procedure 
for all Air Force system programs. 

One of the products of the Definition Phase identified by the DOD was a 
detailed PERT/Cost network "for the development of all items contained in the 
system or portion thereof on which he (the contractor) was asked to bid." A 
list of all end items was to be included and, for each end item, performance 
specifications, cost estimates, and foreseeable technical problems were to be 
presented. In retrospect, the dismayed reactions of the MITRE and SDC staffs 
to these demands upon their technical virtuosity are not difficult to under- 
stand. Detailed end items had never before been systematically identified in 
advance of development for the then current crop of L-systems. Disagreement 
was rampant over the kinds of products which should be included under the 
rubric "software." Did it, for example, include "system training," "human 
engineering," "system evaluation," etc.? No reliable cost estimates for 
computer programming tasks in military command and control systems existed. 
SDC personnel could not see how computer program end items could be identified 
sufficiently at this early phase in the system life cycle to provide a basis 
for cost estimates accurate enough for fixed price or incentive contracts — the 
basic objective of the DOD instruction. 

Considering the computer programming state-of-the-art, the requirement 
to identify "foreseeable technical problems" during Phase IB, was regarded 
by experienced programmers as an unrealistic one. In the early 1960s, 
computer programmers regarded a large part of what they did in military systems 
as research or exploratory development. The DOD requirements implied a more 



highly developed technology. In Phase IB, for example, it was stated that the 
contractors' approach "should be such that the system development to follow can 
be accomplished essentially as an engineering task requiring little or no 
further exploratory investigation." In general, the feeling at ESD was that 
the DOD concept of the system life cycle had the phases in a reverse order 
of emphasis insofar as L-systems were concerned. For L-system development, 
more time should be devoted to system design and relatively less to acquisition. 
They saw the basic problem as "the proper phasing of software with hardware 
design and development." Past experience indicated that requirements for the 
computer programs were usually underemphasized during the early stages of a 
system program. Prior to 1963, for example, it was common for computer 
equipment, displays, and operator consoles to be contracted for before the 
writing of performance-level specifications for computer programs was initiated. 

In spite of such questions, however, the obvious importance of the general 
problem motivated ESD and its supporting not-for-prof its to take immediate 
action. 

The ESD/MITRE/SDC Steering Committee established the third Working Group at 
this point to provide technical support to ESD in preparing its response to 
AFSC, and to provide limited support to the interdivisional team which was 
already in session during the first week in April. Known as the System 
Acquisition Working Group (SAWG) this group led an active and productive 
existence for a period of about six months. Its primary objective was to 
define an acquisition .process which would take into account the unique needs 
of electronic system development as viewed in the ESD management framework. 
The personnel associated with SAWG continued to provide occasional support 
to the interdivisional task force during. that period. However, the factors 
of remote geography, disparities of initial orientations, and inherent 
complexity of the problems prevented the SAWG efforts from being materially 
influenced by the command-wide concepts being evolved at Norton, which sub- 
sequently resulted in the 375-series manuals. 

The SAWG resulted in two published reports which, although issued separately 
by MITRE and SDC, represented complementary treatments of the total system 
problem as viewed at that time in the ESD context. These were: SDC 
TM-(L)-LX-7A/000/00 (Draft), Command Control Software Subsystem Development 
During the Conceptual, Program Definition, and Acquisition Phases, lA August 
1963; and MITRE TM-69, The Electronic Systems Acquisition Process, 31 October 
1963. 

Both reports, in different ways, were something less than finished products. 
TM-69 defined a general process at the total system level, in terms of the 
four subsystems which had been chosen as typical of an electronic system — 
namely, hardware, software, testware, and facilities. While the intent was 
to amplify processes subsequently for the subsystems (except software, which 
was amplified in the SDC report), this intent was never realized. The SDC 



document was published only in draft form. Like TM-69, it contained a variety 
of management concepts and assumptions which later proved to be incompatible 
with the Command-wide 375-series philosophy. However, it did provide, for 
the first time, a comprehensive description of the technical processes 
involved in "software" development which both reflected a breadth of L-system 
experience and established valuable precedents for subsequent work. 

C. DEVELOPMENT OF CURRENT CONCEPTS AND PROCEDURES 

The SAWG activities terminated in late 1963. However, the, Command-wide 
efforts which were destined to result in the 375-series manuals continued, 
with increasingly active participation by ESD. During the early part of 1964, 
the attention of aerospace industry and defense agencies was drawn to the 
newly-revised issue of AFSCM 375-1, which had begun to expand the scope and 
importance of configuration management. Although the immediate impact of the 
manual was on systems and items of equipment, a decision had already been 
reached by Systems Command that the principles of configuration management 
should be extended to cover computer programs. In effect, this was a decision 
that computer programs are a class of system components which should be managed 
like items of equipment — as opposed, specifically, to items of data. 

The decision to classify computer programs with equipment for purposes of 
management was not altogether novel. However, appearing as it did in the con- 
text of the emerging 375-series concepts, it provided a new set of criteria 
and guidelines for approaching questions of computer program acquisition which 
had not been present in the earlier ESD studies. Further studies were clearly 
needed, since it was recognized that the equipment procedures were of 
questionable applicability to computer programs in many areas, not only within 
configuration management but throughout the systems management structure. As 
a result, and because of ESD's primary interest in computer-based systems, ESD 
was designated the lead division for Systems Command to resolve the associated 
problems . 

Since that time, the tasks undertaken have been concerned with developing the 
necessary adaptations of requirements and procedures in the areas of configu- 
ration management, data management, testing, and system engineering, using 
the established manuals covering those areas as points of departure. This 
process actually began in late 1963, as a result of a tentative proposal made 
by the 416M SPO to apply the ANA Bulletin No. 445 to SDC's contract for the 
BUIC II computer programs. Problems posed by the bulletin's equipment-oriented 
language and procedures stimulated SDC to undertake, by way of a counter 
proposal, to develop a special exhibit which would apply to managing computer 
program changes. A series of drafts of the exhibit, initiated in December, 
1963, evolved into a final exhibit which was attached to the BUIC II contract 
in June 1964. 



In the interim, SDC had also been invited by the Technical Requirements and 
Standards Office of ESD to comment on the revised AFSCM 375-1 which was issued 
in draft form in January 1964. Through interest in the general problem, SDC 
initiated a staff study, expanding the BUIC II exhibit effort, which both 
influenced the content of the BUIC II exhibit along lines suggested by 375-1 
and also resulted in a report (SDC TM-1918, 1 June 1964) containing a proposed 
approach to computer program configuration management for general application. 

In January 1965, the Technical Requirements and Standards Office took steps 
to initiate continued studies on a more formal basis. A Software Management 
Committee, composed of representatives from a number of ESD offices, was 
formed to coordinate and direct the efforts, and arrangements were made for 
contract support by SDC. At SDC, the contract work was carried out by a small 
group of the personnel who had been associated with the earlier work in SAWG 
and configuration management.* The contract for the first six-months period 
was sponsored by the 416M SPO, with specific objectives to (1) develop a set 
of AFLC/AFSC Forms 9 covering the information processing data items for a 
typical system program, to fill a vacuum which existed in that area in the 
310-1 manual, and (2) to expand and refine the requirements for computer pro- 
gram configuration management. Requirements incorporated in the BUIC III 
contract, which are the subject of this report, were largely products of that 
effort. 

Continued work by the ESD/SDC team during the next few years was devoted to 
refining those early products, expanding the coverage of 375-series areas, 
and coordinating products with other agencies. Figure 2-1 summarizes the 
events which have been described, and also indicates a number of relevant 
events which have occurred since. The principal documents shown on the chart 
beyond those already discussed are identified and explained briefly below. 

ESD 416M-34 — The BUIC III configuration management exhibit was issued 
before the general exhibit, EST-1, became available. It was based 
on materials resulting from the project described above (SDC 1^-1918/ 
01) , but with a few revisions which (1) had been made in the course 
of subsequent Air Force/industry coordination and (2) were indicated 
for the specific BUIC III applications. 

EST-1 — The ESD Exhibit EST-1 was first issued in May 1966. Based on 
TM-1918/01, it was reorganized into the form of change pages and 
additional exhibits to AFSCM 375-1. 



* This group became the SDC System Management Project, which has been in 
existence since that time, usually at a level of 2-4 members. Current 
members are the authors of this report. 
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EST-2 — ESD Exhibit EST-2 was directed towards the "parent" manual of 
the 375-series, AFSCM 375-4. The exhibit contained some minor 
additions to the 375-4 flow charts of events for a system life cycle 
and a set of supplementary narratives, keyed to the narratives of 
375-4, to clarify application of the manual to ESD systems- The 
exhibit was revised and reissued the following year (1967) in the 
form of ESD Supplement 1 to AFSCM 375-4. 

EST-3 — ESD Exhibit EST-3, "Instructions for Conducting Formal Technical 
Reviews, Inspections, and Demonstrations," was written, by personnel of 
the Technical Requirements and Standards Office to clarify the subject 
topics for the benefit of the program offices. It covers those 
requirements as they pertain to systems and equipment, as well as 
to computer programs. 

TM-3596 — This document, entitled "System Engineering Guide for Computer 
Programs," was initially issued in September 1967 as an SDC Technical 
Memorandum. It was subsequently reissued as ESD-TR-68-1, in March 
1968. As the title suggests, it was directed towards the areas 
covered by AFSCM 375-5. However, it was written in informational 
rather than directive form, to provide a basis for further study 
in relation to the applicability of 375-5 requirements. 

DOD 5010. 19/. 21 — The DOD Directive 5010.19 and Instruction 5010.21 
were issued in July and August of 1968, providing authority for 
the subsequent issuance of military standards 480 and 490 con- 
taining uniform configuration management requirements for use by 
all defense agencies. These events have caused the Air Force to 
initiate revisions of 375-series AFSC manuals to comply with the 
new standards, and will result in some variations in procedures 
and terminology from those described during later chapters herein. 
The conversion has not yet been accomplished at the time this report 
is published. 
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CHAPTER III 
ORIGINS AND DESCRIPTION OF THE BUIC III AIR DEFENSE SYSTEM 

A. INTRODUCTION 

The history of the BUIC system program is treated in two parts in this and the 
following chapter. This chapter reviews the evolution of BUIC as an air 
defense system out of its predecessor systems: the Manual Air Defense System 
and the Semi -Automatic Ground Environment System (SAGE). The major reasons for 
the creation of the BUIC III system are reviewed and the nature of the system — 
its operation and organization — are described. Chapter IV reviews the history 
of SAGE and BUIC II in terms of their acquisition as system programs managed by 
an ESD SPO. 

B. MANUAL AIR DEFENSE SYSTEM 

The defense of the continental United States against air attack was not regarded 
by the military services as a serious problem until after World War II. Two 
events altered military thinking about the need for an effective continental 
air defense: the successful explosion of an atomic device by the U.S.S.R. in 
1949 and the outbreak of the Korean War in 1950 which raised the possibility 
of an attack directly against the United States mainland by manned Soviet 
intercontinental bombers. 

The defense of the continental United States against air attack in the early 
1950s was the responsibility of the Air Defense Command (ADC) of USAF. ADC 
operated an Aircraft Control and Warning (AC&W) network composed of a few 
hundred radar stations in the United States and Alaska.* 

Responding to the pressures of the cold war, ADC's manual AC&W network gradually 
expanded. In cooperation with Canada, radar coverage was extended outside of 
the United States to include the so-called Pine Tree Line, the Mid-Canada Line, 
and the Distant Early Warning (DEW) Line. Radar picket ships and early warning 
and control (radar) aircraft patrolled the approaches to the east and west 
coasts of the United States, thereby extending continuous radar coverage 
hundreds of miles out to sea. The Greenland-Iceland-United Kingdom (GIUK) Line, 
also composed of airborne radars, extended radar coverage across the north 
Atlantic. The Ballistic Missile Early Warning System (BMEWS) was added to 
provide warning against a possible U.S.S.R. ballistic missile threat. 



* A USAF plan in 1947 called for the construction of 411 radar stations at a 
cost of about $4,000,000. 
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The primary functions of air defense were to detect, identify, intercept, and 
destroy the elements of an enemy air attack. The operations required to detect 
and identify aircraft and to control interceptors were carried out in Air 
Defense Direction Centers, the basic units of the AC&W network. These Direction 
Centers (DCs) were located at the surveillance radar stations where air traffic 
was kept under constant observation. 

The basic operations within the DC may be briefly described. Operators watched 
displays of radar echoes on radar scopes for signs of movement indicating the 
flight of aircraft. After detecting an aircraft, the operators would pass its 
range, azimuth, and estimated speed to plotters who marked its position 
relative to local geography of the DC on a large, vertical. Plexiglass board. 
Other plotters maintained the current status of interceptor squadrons, status 
of winds aloft, and status of the air defense system as a whole. The radar 
operators kept the plotters informed of the movements of an aircraft, referred 
to by an assigned track number, as it flew over the DCs geographical area of 
responsibility. Each track was quickly identified from available flight plans 
or was declared unknown if no match with a flight plan could be made. Infor- 
mation on tracks was passed laterally to adjacent DCs and vertically to higher 
headquarters known as Control Centers (CCs) , each of which had jurisdiction 
over several DCs. 

If it became necessary to identify an unknown track visually or to intercept 
a potentially hostile aircraft, the Control Center issued scramble orders 
to a selected interceptor base and then assigned control of the interceptor to 
the DC best located to direct the interception. By following the flight of the 
unknown aircraft and the interceptor on his radar scope and by passing verbal 
instructions to the interceptor pilot via radio, a weapons director guided the 
fighter pilot to a point where visual identification of the unknown aircraft 
could be made, or, if the unknown was determined to be a hostile track, into a 
position to intercept and destroy it. 

Organizationally, each DC was identified as an AC&W Squadron. The commanding 
CC and its several assigned AC&W squadrons comprised an Air Defense Division. 
The several Divisions into which the United States was divided reported to 
North American Air Defense Command (NORAD) headquarters at Colorado Springs, 
Colorado. In 1956, the continental United States was organized into twelve 
Divisions grouped into three Air Defense Forces. 

The Manual Air Defense System suffered from two basic weaknesses: inadequate 
radar coverage and an inability to process with sufficient speed and accuracy 
the large amounts of data which could be expected to accompany any reasonably 
large-scale air attack. The Ground Observer Corps was utilized to attempt to 
fill in the gaps in radar coverage. In response to the problem of data 
processing requirements beyond the system's capabilities, USAF requested the 
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assistance of the Massachusetts Institute of Technology. In 1951, a special 
study known as Project Charles was instituted by MIT. Recommendations stemming 
from Project Charles led to the creation of the Lincoln Laboratory which 
investigated the feasibility of using an electronic digital computer for 
processing radar data. It was this investigation and other similar studies 
which led to the concept of a Semi-Automatic Ground Environment system or SAGE. 

Throughout the growth and development of SAGE, selected elements of the manual 
system have continued to provide a back-up capability for the air defense 
network. 

C. THE SAGE SYSTEM 

SAGE was the first real-time, computerized air defense system in the operational 
inventory of the Air Defense Command. Headquarters, USAF approved its acquisi- 
tion in 1953 and the first site attained operational status in 1958. The 
rationale for the replacement of the Manual Air Defense System by SAGE is reviewed 
in these words:* 

"In early 1950, the military concluded that the manual air- 
defense system in use at that time could not adequately co- 
ordinate use of our improved hardware against the growing 
enemy threat. The capacity of the system was too low; the 
speed with which enemy aircraft could be detected, tracked, 
and intercepted was too slow; and the area over which an air 
battle could be closely coordinated was too small. The problem 
was one of inadequate, nation-wide, data-handling capability: 
facilities for communication, filtering, storage, control, and 
display were inadequate. A system was required which would 1) 
maintain a complete, up-to-date picture of the air and ground 
situations over wide areas of the country, 2) control modern 
weapons rapidly and accurately, and 3) present filtered pictures 
of the air and weapons situations to the Air Force personnel 
who conduct the battle." 

SAGE performs essentially the same operational air defense functions as did its 
manual predecessor: surveillance, identification, and weapons control. SAGE 
is different in one basic respect — it employs electronic digital computers, 
in addition to human operators, to accomplish selected air defense functions. 
While Air Force personnel carry out such tasks as the initiation, identification, 
and monitoring of tracks, the computer programs plot, correlate, and display 
surveillance data. The assignment and commitment of weapons are made by weapons 



* See R. R. Everett, et al . , p. 148, in the bibliography 
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directors, but the central computer continuously tracks the position of 
interceptors and makes the calculations necessary to guide a weapon to its 
target. Man-computer communications flow is shown in Figure 3-1. A key 
characteristic of SAGE, deliberately built into the system, is its semi- 
automatic operations: human operators have the capability to intervene at 
practically any point in automatic processes. All major decisions are made by 
Air Force personnel. 

The SAGE DC contains over 100 console positions, although actual manning at any 
given time varies with traffic load and the military situation. The opera- 
tions of the DC are conducted in several rooms, each responsible for a basic 
air defense function: the Surveillance Room, where radar returnfe and tracks 
are monitored; the Weapons Room, where interceptors and missiles are assigned to 
targets and controlled; the Identification Room, which processes flight plans; 
the Manual Inputs Room, where data received by voice and teletype are inserted 
into the computer; the Command Post, where the Battle Staff may observe the 
course of the air situation or battle on a large wall display; and the Training 
Battle Staff Room, where training and simulation activities are conducted. 

The SAGE Direction Center (DC) is the basic operating unit of the system, 
modeled after its manual DC prototype. However, communications laterally with 
adjacent DCs and vertically to the Combat Centers are primarily digitalized. 
The elements linked to the SAGE DC by both automatic and manual information 
processing capabilities are shown in Figure 3-2. These elements include the 
command element, the SAGE CC; adjacent SAGE and manual DCs; the Army Air 
Defense Command Post with its complement of Nike guided missiles; interceptor 
bases; weather stations; the Air Route Traffic Control Center, which through 
its Air Movements Information Section (AMIS) provides the SAGE DC with flight 
plan information; long range and gap filler radars; radio transmitters linking 
the DC by voice and digital data to interceptor pilots; BOMARC guided missiles; 
and, at various times in the past. Navy radar picket ships and Air Force early 
warning and control aircraft. 

The SAGE system, in 1966, consisted of 14 DCs under the command of 5 CCs, one 
of which (Ottawa) was a combination CC/DC. 

Each SAGE DC contains duplexed IBM digital computers, designated the AN/FSQ-7. 
The two components ensure round-the-clock operations. The following descrip- 
tion of the computer system and computer programs as of 1966 is quoted from 
H. Sackman.* 

**The computer system consists of the central computer, the 
computer programs, magnetic drum buffer storage, six magnetic 



* See H. Sackman, pp. 109-111, in the bibliography. Changes to SAGE have, of 
course, occurred since this description was written. 
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tape units on each side,* and a real-time clock. The central 
computer is a general-purpose,** binary, parallel, single- 
address machine with a 32-bit word length. The magnetic core 
memory originally included 8,000 registers, later expanded to 
69,000. Memory cycle time is 6 microseconds, and average 
instruction operating time is close to 10 microseconds, per- 
mitting up to 100,000 dynamic instructions to be executed per 
second, " 

"There are 12 magnetic drums, each with 12,288 32-bit words, 
for a total capacity of some 150,000 words. The magnetic drums 
are used for storage of the operational program, system status 
information, and for in/out buffer data. A break feature permits 
central computer operations to continue while data are transferred 
from core memory to a terminal device. For example, although 
325 microseconds are required to transfer a word from core to 
tape, only six microseconds of central computer time are used 
during this period for the instruction that initiates the 
transfer. This feature is most important for a real-time 
computing system since considerable time is consumed in input/ 
output searching, waiting and transfers. Separate read-write 
heads permit in/out operations between external sources and 
buffer storage to occur independently of central computer 
operation. This frees the central computer from routine 
service activities and permits it to do more complex jobs more 
closely tied to air defense operations." 

"The operational or real-time program system for the SAGE 
Direction Center contains about 100,000 instructions. It is 
partitioned into some 40 sub-programs to handle specific 
functional areas. Roughly speaking, the operational program 
is approximately equally divided in size between four general 
areas: input /output operations, man-computer communications, 
tracking and weapons control, and miscellaneous areas including 
bookkeeping (updating tables) and simulation. The various Q-7 
support programs involve over half a million instructions. 
They include utility, assembly, recording, data reduction, 
simulation, site production, and JOVIAL compiler programs." 



* Four tape units were standard at SAGE sites. 

** The AN/FSQ-7 was actually designed to incorporate special-purpose features 
for air defense application. 
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"There are additional operational and support programs for the 
AN/FSQ-8* (Q-8 for short) used at higher headquarters at SAGE 
Combat Centers. The Q-8 computer is physically identical with 
the Q-7 computer as it was originally designed with an 8,000 
word core memory. A broader definition of the computer program 
system might reasonably include contractor activities in direct 
support of SAGE training and operations, activities requiring 
the use of SAGE and other computers and associated support 
programs. Under this broader definition, the grand total of 
SAGE-related programs incorporates more than one million com- 
puter instructions." 

The SAGE system suffered from one very critical flaw — it was extremely 
vulnerable in an age of thermonuclear missiles. As so frequently happens in 
the race between offensive and defensive weapons development, USAF had 
committed itself to SAGE before the advent of the ICBM. In addition to the 
question of SAGE survivability, USAF was concerned over what it regarded as 
unexpectedly high operational and maintenance costs. Numerous studies were 
conducted in the 1959-1961 period in an attempt to find solutions to SAGE 
inadequacies. 

In the 1959-1960 period, Super Combat Centers were proposed and evaluated in 
an effort to reduce SAGE vulnerability. The SCCs were conceived as hardened, 
underground sites. However, the SCC concept did not resolve the problem of 
the high cost of air defense. Rather than build expensive underground Combat 
Centers, USAF turned in 1961 to an alternative concept — a decentralized 
dispersed, less vulnerable and less expensive system which would provide a 
back-up capability in the event SAGE facilities were destroyed. 

The Back-Up Interceptor Control or BUIC system was a natural outgrowth of SAGE 
modes of operation. In Mode I, normal air defense operations are conducted 
by the SAGE DCs. If the active computer failed, the standby computer assumed 
operational responsibility. In the event a SAGE DC was completely disabled. 
Mode II operations began. In this mode, adjacent DCs expanded their geo- 
graphical areas of responsibility to cover the disabled sector. Mode III 
referred to an operational situation in which manual or back-up sites assumed 
the task of air defense when their SAGE DCs were inoperative. 



D. THE BUIC CONCEPT 

In 1961, a Task Group at Headquarters, USAF developed the concept of m\ 



* The Q-8s have since been phased out, 
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austere interceptor control system as a back-up to SAGE and submitted their 
plans to NORAD, ESD, and ADC. The cost of the system was not to exceed $100 
million, using off-the-shelf equipment, available communications, and was to 
be acquired in the shortest possible time.* The USAF planners believed that 
the BUIC concept would make a SAGE reconfiguration possible in which the most 
vulnerable DCs would be eliminated and, as a result, USAF would realize 
significant savings in operations and maintenance costs. In January 1962, ADC 
published an operational plan for BUIC and a month later ESD published a 
document dealing with BUIC functional design and performance requirements. 
These early documents recommended the creation of back-up control centers, 
called NORAD Control Centers (NCCs) , which would take over the air defense 
responsibilities in the event SAGE control capabilities were lost. The 
assumption that the NCCs would have a higher probability of surviving a missile 
attack than SAGE DCs was based on the fact that they were to be collected 
with selected long-range radar sites which were not near expected ICBM targets. 

The BUIC system implementation was divided into three phases. BUIC I was 
conceived as an interim system consisting of 27 manual NCCs. This system 
provided an immediate manual back-up control capability comparable to the 
earlier Manual Air Defense System. While some of these BUIC I NCCs will 
remain part of the air defense system, most will be either phased out or con- 
verted into semi-automatic NCCs. BUIC II waS originally planned to have 3A 
NCCs; however budget limitations and other factors reduced that number to 13 
computerized NCCs capable of assuming the SAGE air defense functions. The BUIC 
NCCs do not have the same data processing capabilities as SAGE DCs, but perform 
basically the same air defense operations in a similar, computer-assisted manner. 
BUIC II is currently operational.** The BUIC II system with improved and ex- 
panded capabilities, roughly twice that of BUIC II, is gradually replacing it. 

E. THE BUIC III SYSTEM*** 

BUIC III is a command control system composed of semi-automatic NCCs located 

at selected long-rage radar sites. One, two, or three NCCs are installed 

in an Air Defense Division to serve as a backup to the SAGE DC (see Figure 3-3). 



* See W. S. Melahn, p. 1, in the bibliography 

** A description of the BUIC II system may be found in a System Development 
Corporation document, BUIC II Interface Exercise Manual, TM-2329/000/02 , 
A April 1966. 

*** The information on BUIC III in this section, with the exception of Figure 

3-3, is taken in its entirety from a System Development Corporation document. 
Introduction to BUIC III and SETE, TM-3000/100/02, 1 February 1968. 
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When the DC in a Division is operating, the NCCs in that Division monitor the 
air defense situation to be able to assume air defense responsibility quickly 
in case the DC becomes inoperative. This operational mode, called the Monitor 
mode, is the normal operating state of NCCs. If a SAGE DC becomes inoperative, 
the NCCs in that Division change to an Active mode of operation and assume 
responsibility for air defense operations within the Division. At the NCC, 
BUIC III operators receive displays of filtered and processed data and enter 
instructions at display consoles connected to the BUIC computer. 

A digital, teletype, and voice communications network interconnects NCCs and 
other air defense system elements. Most of the data entering the NCC are in 
digital form and are entersd directly into the computer. 'Other data received 
by voice or teletype may be entered into the computer manually by card inputs. 
The NCC, in either a Monitor or an Active mode, receives a continuous flow of 
data which must be processed and displayed to support its air defense role. 
There is also a continuous flow of data out of the NCC, but the volume of out- 
puts is much less in the Monitor mode than in the Active mode when the NCC has 
active control of air defense elements in its area of responsibility. The 
major elements that a typical NCC interfaces with are depicted in Figure 3-4. 

In the Monitor mode, an NCC maintains a current air picture by means of radar 
data received directly from radar sites and track data received from its 
parent DC (the DC in its Division) or from associate NCCs (other NCCs in the 
same Division) . Radar data come from the same network of radars used by the 
SAGE system. This network includes long-range radars (LRRs) , gap-filler 
radars (GFRs)* and airborne long-range radar inputs (ALRIs) . These data are 
processed and displayed at the NCC in the same manner in either the Monitor 
or the Active mode. An NCC in the Monitor mode, however, has no control over 
the operations at the radar sites • In the Monitor mode, the track data from 
the DC are automatically updated by the BUIC computer program by means of 
radar inputs. This helps to ensure the continuity of tracks in the system in 
the event the NCC has to take over the air defense function. While in the 
Monitor mode, an NCC does not initiate tracks, request height, or identify 
live tracks, nor does it commit weapons. 

When the parent DC becomes inoperative, the NCCs in the Division take over the 
functions of the DC. The Active mode NCCs expand their interfaces and each 
assumes control of a predesignated portion of the Division. External defense 
elements are informed by voice that the NCC is assuming control and circuits 
are established for the flow of information. An NCC replaces its parent DC 
in the lateraltell network of DC communications. One NCC in the Division 
becomes interconnected with the SAGE Combat Center (CC) which is responsible 



* All but one of the GFRs have been phase out. 
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for the direction and coordination of air defense functions in the Region. 
The NCC transmits track data and weapons status information to the CC via data 
link. By voice, the CC provides the Active NCC with early warning information, 
nuclear detonation reports, and information on adjacent Region activity. If 
the CC becomes inoperative, and no DC in the Region remains in operation, the 
Battle Comirander in a designated NCC establishes voice communications with the 
NORAD Combat Operations Center (COC) in order to transmit Region status data, 
battle damage, and track information to the COC and to receive threat infor- 
mation, nuclear hazard data, intelligence data, and battle direction. 

An Active NCC is responsible for all air surveillance and weapons control 
functions in its area of responsibility. Close coordination with radar site 
personnel is needed to maintain the best possible surveillance radar data. 
Height finding radars at these sites are also controlled. Tracks are initiated 
and monitored by air surveillance personnel at the NCC, Identification 
responsibilities are assumed. Flight plan information received from the Air 
Movements Information Section (AMIS) via teletype is inserted into the computer 
by card input for manual and automatic identification processing. Weapons 
personnel commit manned Interceptors or BOMARC missiles against tracks 
identified as Hostile or the tracks are passed to Army Air Defense Command Post 
(AADCPs) for engagement by ADA fire units. Manned interceptors are scrambled 
by means of voice communications between NCC weapons personnel and personnel 
at the Combat Alert Center (CAC) at an airbase. Control of the aircraft is 
ordinarily by means of data link transmitted via Time Division Data Link 
(TDDL) but two-way voice communications are provided by ground-to-air voice 
radio sites. BOMARC control is always via TDDL. 

In the remainder of this chapter, MCC equipment, computer programs, and 
personnel are described briefly. 

1. Equipment 

Data processing and display equipment are installed at each BUIC II NCC to 
support the NCC air defense role. As an integrated group this equipment, the 
AN/GSA-51A Radar Course Directing Group, is capable of performing the calcula- 
tions and data manipulations required by computer programs; accepting and 
transmitting information via digital data circuits; displaying information 
to operators; accepting and processing manually inserted information, operator 
requests and commands; and storing information for subsequent processing. 

a. Data Processing Equipment 

The BUIC III data processor consists of solid state computers, core memories, 
and input/output modules. Their functions and the functions of related peri- 
pheral equipment are described below. 

Digital data computers . Two digital data computer modules interpret and 
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execute program instructions. They operate independently and each is capable 
of controlling the operation of a stored program. 

Core memories . Eight core memory modules provide a high-speed random access 
storage of a total of 32,768 words. 

Controller-comparators . Four controller-comparators or input /output control 
modules control the transfer of data between the core memories and peripheral 
devices. 

Magnetic c^rums . Three magnetic drums are used for the storage of computer 
programs, large blocks of data, and information to be displayed. Each drum 
provides 65,536 words of storage. Three magnetic drum controller-converter 
enable the transfer of data from a display drum to a data display console. 

Message processors . Two message processors receive digital data from external 
sources, store the data, and send them to a controller-comparator. They also 
store output messages from the controller-comparators prior to transmission 
over the output circuits. 

Magnetic tape units . Four magnetic tape units or recorder-reproducers are 
used to record data and store computer programs and simulated data. They are 
made compatible with the controller-comparator modules by means of a recorder- 
reproducer controller. 

Teleprinter set . An on-line teleprinter produces permanent records of selected 
data. This high-speed device can print 120 characters per line in quadruplicate 
at a rate of 600 lines per minute. 

Keypunch . A keypunch machine is used to punch and verify data on 80-column 
cards. 

Punch card reader . An on-line punch card reader is used for the manual inser- 
tion of information to be used by the computer program. 

Teletypewriters . Teletypewriters (teletypes) are used to receive flight plan 
data, winds aloft data, and weather information. They output information in 
both punched paper tape and printed form. 

047 Tape-to-card converter . This device converts the punched tape output by 
the teletype to punched cards which are ready for entry into the card reader. 

Typewriter-punch reader . An on-line typewriter-punch reader provides a means 
of communication between the data processor and maintenance personnel. 
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Status display console . A status display console is provided in the data 
processing area for use by maintenance personnel in controlling and monitoring 
all central and peripheral data processing equipment. 

Simulator group . The simulator group consists of two manual input keyboards 
located in the data display area and one compatibility unit in the data pro- 
cessing area. This equipment can be used to generate and control simulated 
interceptors and to simulate pilot response during a training exercise. 

b. Data Display Equipment 

Data display consoles provide operators with the computer-generated displays 
they need to perform their assigned tasks, a means of selecting the information 
to be presented, and a means of inserting requests and commands into the com- 
puter. Eleven data display consoles are provided in those NCCs having an Air 
Defense Artillery responsibility. All other NCCs have ten. All consoles 
are logically and electrically identical, but they may be assigned to different 
functions by means of manual input card messages. A typical configuration of 
display equipment and assignments is shown in Figure 3-5. The minimum and 
maximum number of consoles that may be assigned to various functions is as 
follows: 



Console Function Min. 

Senior Director 1 

Weapons Director (s) 

Air Defense Artillery Director(s) 

Radar Inputs and Countermeasures 

Officer/Air Surveillance Officer 1 

Air Surveillance Operator (s) 1 

Identification Operator (s) 

Simulation Supervisor 

Flight Path Simulator (s) 

Target Monitor 



Max , 
1 
6 
2 

1 
7 
2 
1 
3 
1 



Situation Display . The data display console contains a cathode ray tube for 
the display of information concerning the air defense situation. It has a 
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♦During simulation exercises, coneole 6 is normalXy 
inanned by a Simxilation Supervisor (SIM SUP), and 
two Interceptor Pilot Simulators (IPSs) who operate 
the simulator group (SG) manual input keyboards. 



Figure 3-5. Typical BUIC III Display Area 
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viewing area 12 5/8 inches square. Symbols and vectors are used to display 
such information as radar data, track information, alarms, intercept points, 
the location of air defense facilities, and geographic boundaries. By push- 
button action an operator may select display expansion levels of two, four, or 
eight in addition to the basic display which covers an area 1500 nautical miles 
square. 

The classes or categories of data displayed on a situation display may be 
selected to be appropriate to the function being performed at each individual 
console. Forty three-position category selection switches are provided on the 
left side of the situation display viewing surface for this purpose. 

Tabular display . A separate cathode ray tube is provided on the data display 
console for the display of data in a tabular format. It has a viewing area 3 
3/6 inches square. Within this area a maximum of 17 columns and 16 rows of 
symbols may be displayed. 

Computer alarms . One visual and one audible computer alarm are provided at 
each console. Activation and resetting of the alarms are under the control 
of the computer program but the audible alarm may also be reset manually by the 
console operator. 

Manual intervention . By means of pushbuttons or switches, an operator may 
enter and transmit information from the display console to the data processor. 
The switch panel on the console contains a twelve-column pushbutton keyboard, 
a rotary heading switch for the insertion of heading information, and an 
activate pushbutton/indicator to transmit information to the data processor. 

Light sensor . Operators are provided with light sensor devices similar to the 
SAGE light guns to initiate requests or commands to the data processor. This 
device is activated by positioning it over a displayed symbol or vector by 
means of a projected circle of light. 

Coordinate data monitor . A coordinate data monitor console is provided for 
the monitoring of radar inputs. This equipment is also called a Random Access 
Plan Position Indicator (RAPPI) . 

2 . Computer Programs 

The computer programs provided for BUIC III were identified and developed as 
four separate computer program contract end items (CPCEIs) , distinguished on 
the basis of system functions as follows: (a) operational functions, (b) 
system exercising functions, (c) utility functions, and (d) computer mainten- 
ance-diagnostic functions . 

a. Air Defense Computer Program (ADP) 

The ADP performs the calculations and data manipulations required for air 
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defense operations. It provides the capability for tracking and identifying 
aircraft, for the scrambling and control of interceptors, and for the transfer 
of information to adjacent facilities or higher echelons. In addition, a 
capability is provided for periodically recording operations data for later 
analysis. For training operations crews, a means of simulating certain inputs 
to an NCC is included in the program. 

b. System Exercise Computer Program (SEP) 

The SEP provides the capability to generate simulated data on magnetic tapes 
to be used as inputs to ADP during air defense exercises and during the 
testing of ADP. It also provides a means for analysis of data recorded during 
exercise and training missions. Summary information may be provided on 
individual operator performance as well as on the performance of groups of 
operators or of the system as a whole, 

c. Utility Computer Program (UCP) 

The UCP is the system used for the production and maintenance of ADP and SEP. 
It is also capable of maintaining itself. In producing a coded system, infor- 
mation is accepted by the program, assembled into a machine language and 
placed on a magnetic tape for later use. These tapes may be changed via the 
UCP to correct errors or to incorporate required changes to the system, 

d. Maintenance Computer Program (MCP)* 

The maintenance computer program is made up of two parts: the backup con- 
fidence-diagnostic (BCDP), and the active confidence-diagnostic (ACDP) . BCDP 
consists of confidence checking and diagnostic testing programs which exercise 
and operate in the backup equipments. This program operates in parallel with 
ACDP or ADP. ACDP operates in the active equipment modules when ADP is not 
running. It provides executive control for parallel BCDP/ACDP operation, 
diagnostic testing, and error analysis. The MCP provides the BUIC III system 
with early malfunction detection and fault isolation to minimize the effect 
of equipment failures on overall system performance. The computer operators 
are kept informed of equipment failures through audible alarms, visible dis- 
plays, or printed output on the typewriter-punch reader or the teleprinter. 

3. Personnel 



Operations and maintenance personnel are stationed at each NCC for the perfor- 
mance of air defense operations. They are divided into functionally oriented 



* MCP is a Burroughs Corporation computer program. 
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groups consisting of a Supervisory Team, an Air Surveillance Team, a Weapons 
Direction Team, a Manual Data Team, a Maintenance Team, and several other 
support teams. The authorized positions which constitute these teams are out- 
lined in the following paragraphs, but the number of operational positions which 
will be manned at any given time will depend on the NCC status and the Division 
air defense conditions. 

a. Supervisory Team 

The Supervisory Team is composed of the Battle Commander (BC) , a Senior 
Director (SD), and a Senior Director Technician (SDT) . The BC is responsible 
for the NCC operation and coordinates with higher echelons of command. The 
BC's position is at a table in the display area adjacent to the SD. He does 
not have a display console, but he may share the SD's console. The SD is 
responsible for air defense functions being performed within the NCC and 
coordinates defense operations with other NCCs. The BC is supported by the 
Weather Officer (WXO) and the Communications-Electronic Staff Officer (CEO) , 
who are located at the same table as the BC. 

b. Air Surveillance Team 

The Air Surveillance Team is responsible for radar input data control, tracking, 
height finding, track identification and coordination with other control 
centers. The team's supervisor is responsible for dual functions. As the 
Radar Inputs and Countermeasures Officer/Air Surveillance Officer (RICMO/ASO) , 
he supervises the action of the air surveillance function, monitors radar 
input data, and coordinates ECCM activities in an ECM environment. He also 
supervises the manual inputs function. The Air Surveillance Technician (AST) 
assists him in these duties. Other team positions include Air Surveillance 
Operators (ASOs) who initiate and drop non-Interceptor tracks, monitor active 
tracing, initiate height requests, and monitor passive tracking; and Identi- 
fication Operators (IDOs) who identify tracks and coordinate flight plan 
information with input agencies. 

c. Weapons Team i 

The Weapons Team is responsible for directing and controlling air defense 
weapons and coordinating activities with other control centers and AADCPs . 
The team consists of Weapons Directors (WDs) and their technicians, and an 
Air Defense Artillery Director (ADAD) and his technician. Army personnel 
are stationed at the NCC to perform ADA functions. 

d. Manual Data Team 

The Manual Data Team is responsible for handling data received by voice or 
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teletype from external sources and entering it into the NCC data processor. 
Normally, the team is composed of three Manual Inputs Operators (MIOs) and a 
Manual Status Clerk (MSC) • The duties of the MSC include the maintenance of 
current status data on a vertical plotting board in the display area. Other 
than this task, the functions of the Manual Data Team are performed in the 
Manual Data/Weather area, which is a 9' by 12' space partitioned within the 
data processing area. This team has no data display console. 

e. Weather Team 

The Weather Team provides weather information to operations personnel in the 
NCC. Weather staff support is given to the Battle Commander. Nuclear fallout 
patterns are developed, and weather forecasts and winds aloft data are prepared 
for the Manual Data Team to enter into the data processor. The Weather Team 
consists of a Weather Officer (WXO) whose normal position is at the EC's table 
in the front of the display area and a Weather Observer (WXOB) who is located 
in the Manual Data/Weather room in the data processing area. 

f . Maintenance Team 

The Maintenance Team is under the direction of the Electronic Computer Main- 
tenance Office (ECMO) who normally supports the SD in the display area. The 
Computer Maintenance Supervisor (CMS) and the Facility Maintenance Monitor 
(FMM) have over-all responsibility for the maintenance and operation of the 
data processing and display equipment. They are assisted by Computer Main- 
tenance Technicians (CMTs) and Display/Input-Output Technicians (D/IOTs). A 
Computer Program Maintenance Team (CPMT) is responsible for monitoring the 
operation of the NCC computer programs and making on-site modifications and 
corrections as necessary. 

g. Target Monitoring Team 

The Target Monitoring Team consists of an Exercise Director and a Target 
Monitor. They are responsible for controlling exercise target aircraft and 
for ensuring the safety of all aircraft during air defense exercises using 
live aircraft. These are not full-time positions but will be filled by 
qualified operations personnel as a supplement to their normal duties. 

h. Simulation Team 

The Simulation Team is responsible for the generation of the simulation environ- 
ment and performance evaluation during a training mission. The only full time 
position is the Training Technician (TT) . Other positions are manned as 
required by available NCC personnel. 



32 



CHAPTER IV 

INFORMATION PROCESSING ELEMENTS 
OF THE SAGE/BUIC II SYSTEM PROGRAMS 



A. INTRODUCTION 

Whereas the preceding chapter presented an outline description of the SAGE/ 
BUIC systems, the purpose of this chapter is to review those aspects of the 
earlier system programs which are relevant to an appreciation of the BUIC III 
experience. As in other portions of the report, the focus of interest is on 
the elments of contractor responsibility which were once known as "software" 
and are presently referred to as the computer program, or information pro- 
cessing, system segment. 

The distinction is being made herein between a system, on the one hand, and a 
system program on the other. By system program we refer to the management 
structure and organizational activities which are involved in the process of 
creating, delivering, and sustaining an operable and supportable system (cf. 
AFR 375-1). The system itself is the collection of equipment, computer pro- 
grams, facilities, procedural data, and personnel which is provided to a user 
for operational emplo3Tnent. Thus, in brief, the system 1,8 the product, while 
the program is the set of time-phased work elements which bring the product 
into being. The point of making this distinction is that the program is basi- 
cally a set of R&D activities, and encompasses many products which are not 
parts of the delivered system. Also, the complete operational system may involve 
elements which were not acquired under a given system program; for example, 
neither of the two BUIC systems could operate without the radar, communications, 
and other elements which were acquired as parts of other air defense programs. 

BUIC II and BUIC III were closely related not only as systems but also as 
system programs. BUIC II provided the direct precedent for defining the scope 
of contractor responsibilities associated with computer program development in 
BUIC III; and BUIC II, in turn, was an outgrowth of the developmental history 
of SAGE. Relevant aspects of those earlier programs are therefore reviewed 
briefly in the following sections of this chapter, to provide a basis for 
understanding the new elements which were introduced in BUIC III. 

B. SAGE 

As the first big computer-based system to be developed at the Hanscom complex 
(now ESD) , SAGE established procedents which have influenced most, if not all, 
of the later L-System programs. Beginning as an R&D project at the MIT Lincoln 
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Laboratory in 1951, it merged into the Experimental SAGE Sector (ESS) in 1953, 
became operational at its first site (McGuire AFB) in 1958, and has continued 
to grow and change to the present time. Thus, its history covers the time 
span during which the data processing arts have had their growth, essentially 
from a state of infancy, to their current and rapidly expanding roles of 
importance as elements of military system programs. SAGE was directly re- 
sponsible for much of the growth; in the late 1950s, more than half of the 
nation's computer programmers were still associated with the program. 

It was a natural result of its close association with the evolving state-of- 
the-art in data processing that SAGE was adopted as a "model" for command and 
control system development. Constant changes in configuration were its most 
distinguishing characteristic. Factors leading to the changes included the 
advent of new weapons, equipment, and tactical/strategic doctrines, as well as 
improved computer programming techniques. The concept of "evolutionary de- 
velopment" was derived from that experience and generalized as a guiding 
principle for command and control system programs. This principle has a variety 
of facets which are closely in conflict with current objectives to procure 
systems as items having pre-defined performance, schedules, and costs. Hence, 
it provided some of the basis of early resistance at ESD to the new systems 
management philosophy (see Chapter II) and the feasibility of developing 
computer programs, in particular, under conditions of fixed performance, cost, 
and schedule requirements has continued to be a matter of debate. 

The evolution of SAGE computer programs occurred in a variety of forms, but 
notably through the development of a series of "SAGE Models". The first 
operational model, which was installed at the New York Air Defense Sector at 
McGuire AFB in 1958, was based on the Model Zero that had resulted from the 
experimental developments at Lincoln Laboratory- Subsequent models incorpora- 
ted various computer program improvements, but were mainly in response to 
requirements for new capabilities or to changes in system/equipment configura- 
tion and interfacing weapons. By the time the BUIC II program was initiated 
in 1962, Model 9 was operational and a next was in the planning stages. The 
last model was never implemented, although new "versions" incorporating exten- 
sive changes have continued to appear at the rate of 3 or 4 per year since 
that time. 

Each new model or version represents a complete re-cycling of the computer pro- 
gram acquisition process through its 6 phases of analysis, design, development, 
testing, installation, and maintenance. The process naturally emphasizes the 
added or altered operational capabilities, but it also typically involves 
altering other parts of the "basic" computer programs to avoid exceeding fixed 
storage constraints and compensate for interactions. The analysis phase in- 
volves conducting studies of feasibility and trade-offs, and defining firm 
requirements to be incorporated in the new model or version. The design phase 
consists of both (1) "operational design", which results in the precise 
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formulation of performance-level requirements, and (2) design of the computer 
program changes to meet the requirements. Development encompasses the coding, 
testing, and debugging of individual computer programs, while the test phase 
is devoted to assembly testing of functionally-related elements, checkout of 
the integrated computer program system at an operational site, and the per- 
formance of tests under live conditions. During installation, site-specific 
adaptation data are incorporated and the new version is checked for operation 
at each site. Once operational, a maintenance* phase begins and continues for 
the life of that model or version, which consists of (1) malfunction diagnosis 
and error correction and (2) incorporating minor or emergency modifications 
on-site, pending appearance of the next complete new version. 

Involvement in SAGE computer programming by the organizational unit of RAND 
Corporation which eventually became SDC was an outgrowth of earlier work in 
the field of operational readiness training for personnel of the manual air 
defense system. The capability known as the Manual System Training Program 
(MSTP) had been developed and put into operational use in the manual air 
defense system during the period of 1954 to 1957. This capability consisted 
of a complex set of simulation equipment, materials, and procedures which could 
be used at the operational sites, permitting personnel manning the air defense 
network to conduct realistic exercises against simulated air attacks. Following 
the introduction and use of MSTP, requirements were established for a similar 
capability to be provided in SAGE. The result was the SAGE System Training 
Program (SSTP), which was developed and implemented concurrently with other 
SAGE elements. And like the other elements, SSTP underwent a continuing series 
of evolutionary refinements and changes. 

The close association of computer programming and training was a natural result 
of their mutual technical concern with and dependence on the detailed pro- 
cedures involved in air defense operations. Also, in SAGE, special computer 
programs emerged as important elements of the on-site SSTP capability; and 
still others were developed to generate the synthetic input data, aids, and 
materials for distribution to the sites. 

As integral parts of the computer program and training development activities, 
extensive documentation was associated with the SAGE models and SSTP problems. 
These documents included reports of special studies and analyses, operational 
specifications for the computer programs, positional handbooks for the SAGE 



* Use of the term "maintenance" for these activities is commonly accepted, 
although some other term might be preferable in the interest of avoiding 
confusion with equipment usage; for example, "modification" is closer to the 
point, since the indicated action in cases of malfunctions or other deficien- 
cies is always to alter the original configuration, rather than to restore 
or maintain it. 
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operational personnel, SAGE System Description, adaptation data, and 
positional guides, manuals, and other materials for SSTP. 

C. BUIC II 

The conceptual studies leading to the choice of a system configuration for 
BUIC II were carried out principally during 1961. The document which served 
as a system specification, entitled BUIC System Functional Design and Perfor- 
mance Requirements, was issued as ESD Document No. 416L-1 in February 1962. 
There followed a period of negotiations, resulting in the award of contracts 
to two associate contractors prior to mid-1962. These were: Burroughs Corpora- 
tion, for system computer and associated equipment; and System Development 
Corporation, for computer programs. 

Although the initiation of BUIC II thus preceded the appearance of the DoD 
Draft Instruction on Program Definition by about a year, pressures for firm 
planning of costs and schedules were in evidence from the outset of the program. 
The equipment contract was negotiated on a fixed-price basis; and the equipment 
schedules and constraints soon became the pacing factors in the program as a 
whole. At that time, ESD studies which led to the MITRE TM-3551 (see Chapter 
II) were just beginning, and little or no guidance was available to the SPO 
for planning the management of computer program acquisition, other than that 
which could be supplied by MITRE and SDC as the program progressed. 

At SDC, SAGE experience provided a ready basis for defining the major technical 
products. However, in SAGE, those products had come into being gradually, 
through the "evolutionary" processes of experimental developmental and successive 
modifications based on use under operational conditions. Such factors as the 
brief and fixed timing of acquisition, concurrency of equipment development, 
close schedules, and the general austerity of funding which characterized BUIC 
II throughout its course were novelties. Hence, the scope and content of work 
elements for the computer program contractor had to be worked out for the first 
time in BUIC II under conditions which were significantly closer to those of 
current system programs than had been true earlier. The resulting elements 
are outlined briefly below. 

Operational Design 

As it has been defined traditionally in SAGE and BUIC, this is the set of 
activities which results in completed "operational specifications" for the 
computer programs. As a level of technical work, it is comparable in a 
general way to the system engineering effort which a contractor would carry 
out to produce Part I specifications for contract end items, although with 
some discrepancies. It encompasses such component elements as: analysis of 
user requirements; study of alternative solutions and trade-offs; development 
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of detailed operating descriptions of computer functions, controls, and 
displays; derivation and selection of mathematical formulas and equations; 
specification of interconnections between console switches and computer (Vari- 
able Display Equipment); and writing of the operational specifications. 

Computer Program Design, Development, and Test 

Although not known as contract end items at that time, elements of the BUIC 11 
computer program system were separately identified as: operational; utility 
and support systems, including an automatic adaptation capability; Field Site 
Production System (FSPS) for on-site generation of Problem Input (PI) tapes 
used in system exercising; and the BUIC Data Reduction System (BUDR) for a 
variety of uses in testing, checkout, system exercising, and operational data 
reduction. The development process for each of these elements was divided into 
a series of phases, which were identified in one BUIC II work statement as 
follows: 

a. Program Design: allocation of functions to subprograms; 
allocation of drum and core storage; design of control 
and timing subprograms. 

b. Program Costing: determination of machine instructions, 
display slots, and programming personnel requirements. 

c. Subprogram Design and Coding: detenriination of logic, 
design of internal and external communications, and 
refinement of storage allocation. 

d. Simulation Testing: initially, testing of subprograms 
(parameter testing) to verify conformance with subprogram 
design; subsequently, testing at successively higher levels 
of assembly to verify conformance with operational specifi- 
cations, using simulated inputs. 

e. Documentation: preparation of technical documents describing 
design, data structure, testing accomplished, and other 
information for using and modifying the computer programs. 

f. Retrofit: incorporation of changes to meet new requirements. 

g. Program Maintenance and Error Correction: diagnosis of 
malfunctions and correction of errors; maintaining cor- 
respondence of computer programs at different locations; 
maintaining technical documentation. 
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Program System Testing (PST) -^ 

Following the installation of system equipment at an operationally configured 
facility, the computer programs resulting from in-plant development were 
installed and subjected to extensive testing, using both simulated and live 
inputs. In BUIC II, this PST activity was initiated at the BUIC Evaluation 
Facility (BEF) , located at Hanscom Field, and was later continued at the first 
operational site at North Truro, Massachusetts. Category II tests were also 
accomplished, to some degree concurrently, at those same locations. 

Data Collection and Processing 

This activity encompassed the collection of geographic, unique-to-site, and 
weapons characteristics data, as well as processing, verifying, publishing, 
and maintaining the data. 

Personnel Subsystem (PS) 

BUIC II was the first system program at ESD to be materially affected by the 
concept of the personnel subsystem, which had become established as Air Force- 
wide policy through the issuance of AFL 375-5 , Planning and Programming' for 
System Personnel, in October 1961.* The spectrum of PS requirements defined 
in AFL 375-5 was set forth in the BUIC II specification issued in February 
1962, although stated at a very general level. They posed a variety of problems 
which were new in the ESD system context and which were also somewhat foreign 
to the PS concept itself because of the fact that PS, like other systems 
management elements which were yet to appear, had originated with aircraft and 
missiles. There ensued a period of study to define implementing methods which 
would be suitable to the requirements of. an information system, and to identify 
workable allocations of responsibilities to the SPO, MITRE, and the two associate 
contractors. As a result, BUIC II preceded BUIC III as a "test bed" for the 
requirements in this area as they relate to computer program acquisition. 
Elements assigned to SDC as PS contract activities are summarized briefly 
below.** 

a. Personnel-Equipment Data (PED) . PED was only partially 
implemented in BUIC II because of the general austerity 
of funding. Information was compiled and maintained by 
each contractor for internal purposes only, at the level 

of whole documents rather than as individual data elements. 

b. Human Engineering, Detailed task analyses for operating 
system personnel were required as a basis for design of 



* The Air Force letter was later superseded by AFR 30-8. 

** An expanded description of the PS adaptations arrived at in BUIC II is con- 
tained in SDC TM-1494, The Personnel Subsystem Program for an Information 
System, dated 23 September 1963. 
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display format/content, switch functions, and special 
computer program functions, and to provide the basic 
data for positional handbooks and QQPRI. 

Qualitative and Quantitative Personnel Requirements 
Information (QQPRI) • Position description information 
was prepared for the operator, siirulation/exercising, 
and computer programming support (Blue Suit) personnel, 
for input to the system QQPRI report which contained 
similar information for equipment maintenance and other 
personnel supplied by the hardware contractor. 

Personnel Subsystem Test and Evaluation (PSTE) . Con- 
tractor support was provided to develop plans and assist 
in conducting PSTE during Category II system testing. 

System Exercising for Training and Evaluation (SETE) . 
Although not Included as a recognized element of the 
general PS structure, SETE was so classified in BUIC 
because of its emphasis on training. Development of 
the SETE capability in BUIC IT was based on the SAGE 
model, SSTP, but with the addition of requirements per- 
taining to use for evaluating system functions and sub- 
functions as well as training and evaluating personnel . 
Tasks required to accomplish the development included 
the following variety of elements: 

(1) analyses of training and evaluation requirements; 

(2) specification of exercising modes and configurations; 

(3) identification of performance/design requirements 
for special equipment for problem generation, sim- 
ulation, communications, and recording; 

(4) specification of computer programs for data 
reduction and recording; 

(5) development of computer programs for on-site 
generation of simulated aircraft tracks; 

(6) development of problems for system-wide exercises, 
together with associated aids and materials; 

(7) development of manuals and guides covering the 
organization, planning, and conduct of system 
exercises. 
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Configuration Management 

The circumstances under which configuration management was introduced in BUIC 
II were mentioned earlier, in Chapter II of this report. At the time the con- 
figuration management exhibit was placed on contract (June 1964), the documen- 
tation to be controlled either was already completed or was in advanced stages 
of development. Hence, the exhibit merely identified the specific documents 
which constituted the baselines, in terms of their contractor document numbers, 
titles, and dates, rather than specifying their format and content. The base- 
lines identified were (1) operational specifications which had been written 
for the operational field-site problem generation (FSPS) , and data reduction 
(BUDR) computer programs, and (2) a series of user manuals which was in process 
of being completed for the utility and support elements. Also, as compared 
with subsequent exhibits, there was no attempt to identify a FACI or establish 
product configuration baselines. In other respects, the control and document 
maintenance procedures were closely similar to those which were later adopted 
for general use, although with a number of minor differences in the names and 
formats of control and maintenance forms. In June 1964, the contract identi- 
fied the following specific activities under a task entitled "Configuration 
Management" : 

a. Prepare Change Proposal (CP) forms for all proposed changes 
to the established computer program baselines. 

b. Prepare Change Recommendation (CR) forms for any system item, 
when considered necessary because of conditions affecting the 
computer program developments. 

c. Provide summary cost estimates for each Class I CP. 

d. Publish, maintain, and revise the Configuration Index. 

e. Publish and update the Configuration Management Status 
Report. 
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CHAPTER V 
BUIC III PROGRAM REQUIREMENTS AND EVENTS 



A. INTRODUCTION 

The Air Force was authorized by the Defense Department to proceed with the 
BUIC III program in November 1964 . This event had been preceded by a variety 
of studies involving ADC, ESD, MITRE, SDC, Rand, and others, as a result of 
which the BUIC III concept had been selected from among several alternatives. 

As a fundamental aspect of the BUIC III concept, the program was initiated as 
a major "modification" of the BUIC II system, rather than as a completely new 
system program, and prior to the time that BUIC II was scheduled to be com- 
pletely operational. Thus, it readily became an extended and modified effort 
of the same agencies, including the same principal contractors (Burroughs and 
SDC) who had been involved in the development of BUIC II. A Definition Phase 
was not required since estimated costs were below the stated DOD thresholds. 
The draft system specification was developed during the first half of 1965 by 
ESD/MITRE, with some support by the contractors, following which formal con- 
tract work initiating system acquisition began at the outset of FY 1966 (July 
1965). 

The BUIC III system specification was the first to be produced at ESD which 
attempted to conform to the guidance established in Exhibit I of AFSCM 375-1, 
which had been issued in essentially its present form* and coordinated among 
the AFSC divisions during the preceding year. The specification defined 
performance/design and test requirements for the system, including require- 
ments for the system exercising capability**, and allocated requirements among 



* The January 1964 draft, which was issued for official use in June 1964, was 
a major expansion and revision of an earlier version of AFSCM 375-1 which 
had appeared in June 1962. 

** The label carried by this capability in both BUIC II and BUIC III is System 
Exercising for Training and Evaluation (SETE) . Although often thought of 
as part of the computer program system segment because of SDC's responsi- 
bility for its extensive computer programming, problem generation, and 
personnel subsystem aspects, the capability is more properly treated — as 
it was in the BUIC III System Specification — as a system-level set of 
requirements. Its impact on the system includes, for example, additional 
capacity of the operational computer, as well as special consoles, communi- 
cations, simulation equipment, and personnel. 
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the five system segments: 

Data Processing and Display System Segment 
Maintenance Computer Program System Segment 
Operational Computer Program System Segment 
Building and Facilities System Segment 
Communication System Segment 

Major developmental work to be performed by the two associates system con- 
tractors was defined by the first three of these segments. The equipment and 
maintenance computer program segments were assigned to Burroughs, while 
activities associated with the operational computer program were assigned to 
SDC. 

To provide a basis for discussing experience with the systems management tech- 
niques, the following two sections are devoted to outlining the elements of 
work which comprised SDC's system segment, and describing the ways in which 
these were influenced by the new management procedures and requirements. Sub- 
sequently, in the final section of this chapter, an attempt will be made to 
present a factual summary of events as they actually occurred during the pro- 
gram. 

B. THE OPERATIONAL COMPUTER PROGRAM SYSTEM SEGMENT 

The basic items to be developed and delivered under the contract were three 
computer program contract end items (CPCEIs), which were identified as: 

ADP — Air Defense Computer Program 
SEP — System Exercise Computer Program 
UCP — Utility Computer Program 

It has been mentioned earlier (Chapter IV) that these correspond closely, with 
regard to general computer program types and functions, with the computer pro- 
grams which had been required in both SAGE and BUIC II. 

In addition to the CPCEIs, important items of deliverable data included the 
computer program specifications, positional handbooks, user manuals, exercising 
manuals and guides, and system exercise problem inputs and materials. For the 
most part, these items had precedents in BUIC II (cf. preceding chapter). 

The technical activities were also similar to those which had been carried out 
under the BUIC II program, except for the pervasive influence of BUIC III 
management innovations. These included: operational design; computer program 
design, development, and testing; development of system exercising problems, 
aids, guides, and materials; and performance of associated personnel sub- 
system tasks. In addition, SDC became responsible eventually for the 
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management and conduct of the Category II test program. 

However, essentially all of these activities were affected in varying degrees 
by the procedures and guidelines which had been developed by the SDC Systems 
Management Project during the first six months of FY 1965 (see Chapter II). 
That effort had been funded by the 416M SPO, with the explicit intent to 
acquire products which could be applied and tested in the BUIC III program. 
Also, the SPO was actively represented* in the ESD Software Management 
Committee which reviewed and coordinated the effort during that phase of the 
project. As a result, it is of note that some of the specific new requirements 
were influenced by known characteristics of the BUIC III system. This was 
considered to be a desirable circiamstance by the committee members, who 
regarded the development of generalized L-System procedures to be the primary 
objective, but felt at the same time that the realism of concepts would be 
materially enhanced to the degree that they could be related to the parti- 
culars of an actual program. Thus, BUIC III became a "test bed" for guiding 
the initial formulation of many of the concepts, as well as for subsequently 
evaluating the experience with their application. 

The procedures applied in BUIC III consisted of adaptations to computer pro- 
grams of established Systems Command procedures, principally in the areas of 
data management (AFSCM/AFLCM 310-1), configuration management (AFSCM 375-1), 
and testing (AFSCM 375-4; AFR 80-14). Except for formal design reviews, the 
program was not specifically influenced by the concepts and requirements of 
system engineering management as set forth in AFSCM 375-5. The specific doc- 
uments which governed the application of these requirements in BUIC III were: 
(1) AFSCM/AFLCM 310-1, together with AFLC/AFSC Forms 9 which were then largely 
ESD-unique, (2) the BUIC III configuration management exhibit, ESD-416M-43**, 
and (3) the contract statement of. work, which incorporated certain require- 
ments in the computer program testing area, in particular, which were not 
covered elsewhere. During the first year of this contract, a few of the Forms 
9 were published in 310-1, and such other documents as ESD Exhibit EST-2 
(later superseded by ESD Supplement 1 to AFSCM 375-4) and ESDP 375-10 (sub- 
sequently ESD Exhibit EST-3) appeared. 

The following subsections describe the requirements which were introduced in 
BUIC III. Most of these were novel, in the sense that they did not exist 



* By Lt. B. Capehart 



** The exhibit was based upon the same materials which were issued a few weeks 
later as ESD Exhibit EST-1. Except for being a self-contained document 
(whereas EST-1 is in the form of change pages to AFSCM 375-1) , differences 
in content as compared with EST-1 are negligible. 
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in the same form in BUIC II. For convenience. Table 5-1 presents a summary 
list which identifies the particular aspects of novelty as compared with 
earlier practice. 

C. SYSTEMS MANAGEMENT INNOVATIONS 

Data Management 

The use of a Contractor Data Requirement List (CDRL: DD Form 1423) was 
initiated in BUIC III, together with the application of Forms 9 governing the 
preparation of identified data items. The applicable Forms 9 included a set 
which had been developed under the Systems Management Project to cover items 
which were judged to be of typical interest in ESD systems, but which had not 
previously existed in AFSCM/AFLCM 310-1. As listed below, these items were 
initially ESD-unique; nearly all have since been published in Volume II of 
310-1: 

Positional Handbook — Information System Operational Personnel 
(ESD 178) (H-109*) 

Contract End Item Detail Specification (Computer Program) , Part 
I (ESD 236) (C-132*) 

Contract End Item Detail Specification (Computer Program) Part II 
(ESD 237) (C-133*) 

Category I Test Plan (Computer Program), (ESD 261) (T-103-1*) 

Category I Test Procedures (Computer Program), (ESD 262) (T-103-1*) 

Category I Test Report (Computer Program), (ESD 263) (T-118-1*) 

Exercise Conduct Manual (ESD 281) (Q-125-1*) 

Synthetic Inputs Operator Guide (ESD 282) (Q-123-1*) 

Evaluation Manual (Information System Exercising Personnel), 
(ESD 283) (Q-124-1*) 

Human Operator Task Analysis for Information Systems 
(Q-] 5-18.0) (Q-118-1*) 

Training Needs/Exercise Requirements Analysis (Q-26-18.0) (Q-119-1*) 



* Refers to the data item number (DD Form 1664) in Volume II of AFSCM/AFLCM 
310-1 as of 30 August 1968. 
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Table 5-1. Summary of Computer Program Management 
Requirements Introduced in BUIC III. 





ITEM 


PRECEDENTS 


NOVEL ELEMENTS 










DATA i 
MGMT 


AFSCM/AFLCM 310-1 
Requirements 


Some Data Items 


Use of CDRL 

Specified Format /Content 

Some Items 


! — 










Part I Specification 


Ops Specs 


Quality Ass.; Interfaces 
Utility Coverage 
Spec. Conventions 


H 


Part II Specification 


TM-1333 

User Manuals (U) 


Comprehensive Coverage 
Up-to-date Accuracy 




M 

B 

c:) 

M 


Control 

& 
Accounting 


CP 

CP (II) 

CR 

Conf. Index 

Status Rpt. 


Part I 
^Only 


ECP 

CR 

SCN 

SCL 

EICC 

Conf. Index 

Ch St Rpt 

V.D. Doc 


Part I 
& 
' Part II 




FACI 


^~ ^ ~ 


Formal Audit 

Reissue 

PCB 










CO 

1 


PDR 


Informal 


Formal Review 
Minutes 




M 
CO 

Q 


CDR 


^^"^ 


Formal Review 
Incremental (w/PQTs) 
Minutes 


' 








H 

CO 
H 
M 


Part I Spec (Sec 4) 
Cat I Test Plan 
Cat I Test Proc. 
Cat I Test Reports 
PQTs 
FQT 

CPT&E 


jP&A Testing 

Ipst 


Required (ESD 236) 
Required (ESD 261) 
Required (ESD 262) 
Required (ESD 263) 
Required 
Required 

(no change) 
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Evaluation Needs/Exercise Requirements Analysis 
(Q-24-18.0) (Q-117-1*) 

Exercising Capability Implementation Plan 
(Q-27-29.0) (Q-120-1*) 

Minutes of Formal Reviews and Inspections 
(ESD 289) (C-131*) 

Users Manual (Computer Program) , (ESD 290) (H-110*) 

Version Description Document (Computer Program) 
(ESD 291) (C-135*) 

Configuration Index (Computer Program) 
(ESD 293) (C-136*) 

Change Status Report (Computer Program) 
(ESD 293) (C-137*) 

System Exercising Problem Package 
(Q-28-56.0) (Q-121-1*) 

System Exercising Problem Agreements Document 
(ESD 295) 

It was indicated earlier that most of the above items, as well as others 
listed in the CDRL which were not unique, had precedents among deliverable 
items of data which had been prepared in BUIC II. However, the CDRL pro- 
cedures as such, a few of the data items (notably, in the test area), and a 
number of specific format/content requirements were introduced for the first 
time in BUIC III. 

Specifications 

The Forms 9 for CPCEI Detail Specifications, both Parts I and II, were derived 
from studies which had attempted to take into account a complex of considera- 
tions posed by (1) the general characteristics of uniform specifications as 
embodied, e.g., in Exhibit II of AFSCM 375-1, (2) previous practices in doc- 
umenting computer program developments, as exemplified in such systems as 
416L, 416M, 465L and 473L, and (3) the similarities and differences between 
computer programs and equipment which relate to purposes of configuration 
management. Hence, they involved a number of novel elements in BUIC III, as 



* Refers to the data item number (DD Form 1664) in Volume II of AFSCM/AFLCM 
310-1 as of 30 August 1968. 
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compared with the "specifications" which had been written previously for 
BUIC II. The major differences may be sunmiarized as follows : 

1. The Part I Specification is basically a performance-level 
specification, as were the earlier "operational specifications 
(ops specs)" which had been in use for SAGE and BUIC II. How- 
ever : 

a. Ops specs had previously been written and provided to 
the user only for operational and support, but not for 
utility, computer programs. Hence, the requirements 
to write a performance-level specification for utility 
elements (including the compiler) was completely new. 

b. The ops specs had not contained a quality assurance 
section comparable to Section 4 of the Part I 
specification. 

c. Interfaces and, in general, the specific identification 
of detailed inputs and outputs for each functional 
element of the CPCEI, had not been systematically in- 
cluded in ops specs. Equipment interfaces, as well as 
personnel operating procedures, had been included in the 
ops specs, however, often in the form of detailed 
narrative descriptions of system operational sequences 
and conditions. In addition, a separate set of documents 
known as "Variable Display Equipment Specifications (VDE 
specs)" had been written, defining functions and labels 
of console switches and detailing the recommended wiring 
of console switches to computer storage locations - 

2. Essentially all of the computer program documentation requirements con- 
tained in the Form 9 for the Part II specifications were based 

upon elements which had previously existed in some form, and 
for some computer programs. However, the requirement had not 
previously existed to provide all of those elements for each 
CPCEI, and at one point in time. Some of the elements, e.g., 
flow charts, had been prepared at early phases of the develop- 
ment process, but had not been prepared at detailed levels for 
all components of each computer program. Nor had the design 
description material generated during developmental phases, 
including flow charts, been revised and updated on a con- 
tinuing basis to reflect actual design decisions resulting 
from the processes of coding, testing, and ongoing changes. 
Thus, the requirement to prepare the type of comprehensive 
description described in the Form 9 for a Part II CPCEI 
specification, which would be accurate in all respects at 
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the time of FACI for each CPCEI, represented a major new element 
in the BUIC III effort. 

Configuration Management 

Most of the procedures and standard forms used for computer program configura- 
tion control and accounting had been developed during the period of 1963-1964, 
and had been used in BUIC II. Only a few changes of a minor nature were made 
in the forms used in BUIC III, which conformed in all essential respects to 
those currently described in ESD Exhibit EST-1. These were: Engineering 
Change Proposal; a Change Report form for reporting Class II changes; Speci- 
fication Change Notice; Specification Change Log; End Item Configuration Chart; 
Configuration Index; Change Status Report; and Version Description Document. 

In BUIC II, traditional operational specifications (ops specs) comprised the 
baselines, which were established and formally maintained only at that 
level — i.e., of performance, rather than at the level of detail design. 
In fact, the controls adopted in BUIC II were largely based on controls which 
had been exercised by ADC to govern performance-level changes in the SAGE 
system for some years previously.* Format and content requirements for the 
ops specs had not been contractually specified. These were replaced in BUIC 
III by the Part I specifications, with new format/content elements as described 
above, which became approved and controlled as Design Requirements Baseline 
documents. In addition, the Part II specifications were also required as 
deliverable items under the contract, and were scheduled to be established as 
Product Configuration Baselines as a result of First Article Configuration 
Inspection (FACI) . 

Thus, while configuration control at the performance level had been introduced 
earlier, configuration management procedures were expanded in BUIC III to 
encompass uniform computer program specifications, FACI, and control of base- 
lines at the product configuration level. 

Design Reviews 

The Preliminary Design Review (PDR) and the Critical Design Review (CDR) 
involved formal meetings and reviews of computer program design data which had 
no direct precedents in earlier systems. Although specific design questions 
had occasionally been matters of interest to the SPO's GSE/TDC, MITRE, during 



* ADCM 55-32, Processing Adaptation Data, Computer Program, and Related Equip- 
ment Changes in SAGE, which has been updated a number of times since, was 
issued as early as January 1961. 
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BUIC II, the ESD SPOs had not, in general, been able to acquire and maintain 
the capability to monitor contractor computer programming efforts at the 
technical design level. At SDC, the entire processes of computer program 
design, coding, and testing had been carried out, of necessity, as "in-house" 
contractor responsibilities. Thus, the objectives, functions, and conduct of 
PDRs and CDRs became matters with which both the SPO and contractor personnel 
gained their first actual experience in BUIC III. The PDR was scheduled as 
a single review of each CPCEI (with minor variations noted in a later section 
of this chapter), concerned with the design approach to the CPCEI as a whole. 
The CDR for each CPCEI was planned to be carried out in increments, con- 
currently with Preliminary Qualification Tests (PQTs) , as a review of the 
completed design of CPCEI components undergoing PQT. 

Note: ESD Exhibits EST-1 and EST-3 emphasize CDR 
as a review to be conducted at the time when de- 
tailed flow charts are available for the computer 
program components, prior to the start of coding. 
However, EST-1 also allows for the variation chosen 
for BUIC III, in which the review was conducted after 
coding of the component undergoing review was com- 
pleted. 

Category I Testing 

Prior to BUIC III, computer program testing associated with the earlier air 
defense systems had been accomplished by the contractor without formal involve- 
ment by the SPO. The test process included in-plant testing of components as 
a normal adjunct to coding and assembly operations at SDC. In addition, it 
also involved the conduct of a comprehensive checkout of the computer program 
at the Category II test site following installation of equipment and computer 
programs and prior to the formal beginning of Category II tests. This check- 
out operation was known as "Program System Testing (PST)". It is considered 
to be an important and necessary extension of the development process for 
operational computer programs, in particular, since the Category II site 
normally provides the first opportunity to exercise the complete set of com- 
puter programs in the operational equipment configuration. 

The Category I test program formulated for BUIC III consisted of the elements 
which are now described elsewhere, in varying degrees of detail, in such doc- 
uments as ESD Exhibit EST-1, ESD Supplement 1 to AFSCM 375-4, SDC TMs 3361 and 
3596, ESD-TR-68-1, and associated Forms 9 in the Test Category of AFSCM/ AFLCM 
310-1, Vol. II. The major elements are summarized briefly below: 

1. Category I Test Plan. A test plan was required for each of the 
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three CPCEIs, These plans were based on the Form 9 (ESD 261), 
but were also influenced by a variety of specific circumstances 
and agreements reached with the SPO, relating to both structure 
and content. 

Classes of Testing, By analogy with the types of Category I 
testing which have been established for equipment CEIs, re- 
quirements were formulated under the following three categories; 

a. Computer Program Test and Evaluation (CPT&E) . In 
general, CPT&E encompasses the testing which a con- 
tractor normally conducts as an integral part of his 
design and development effort. Thus, this category 
subsumes essentially all of testing which had been 
typical of the preceding systems, as described above. 
Major requirements formally recognized by the SPO 
with respect to CPT&E pertain to in-plant use of 

GFE computing equipment, and to utilization of the 
Category II test site for computer program install- 
ation and checkout (formerly PST; cf, above), 

b. Preliminary Qualification Test (PQT) . PQTs are con- 
ceived as interim tests which are intended to serve 
the primary purpose of demonstrating contractor pro- 
gress towards meeting design objectives for a computer 
program CEI. As expressed in current guidance documents 
(e.g., ESD Exhibit EST-1) , they are oriented towards 
verifying portions of the CPCEI prior to formal quali- 
fication, but are not expected to accomplish the quali- 
fication as such. This concept was introduced for the 
first time in BUIC III, without benefit of either 
amplifying guidance or experience on the part of the 
SPO or contractor. As a result, PQTs were required 

and performed in BUIC III with somewhat greater emphasis 
(and difficulty) than might normally be expected. PQTs 
were scheduled to test/demonstrate essentially all re- 
actions within each function of the ADP and SEP CEIs, 
prior to formal qualification tests. Each PQT was 
combined with an incremental CDR of design data per- 
taining to the tested components. No FQT (Formal 
Qualification Test) and PQTs only for two of its 
many functions were conducted on the utility com- 
puter program (UCP) . To save time and cost, com- 
plete Category I testing was not required for UCP, 
which could reasonably be considered qualified 
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through its extensive use during the program in 
developing ADP and SEP. 

c. Formal Qualification Test (FQT) . FQTs were required 
for the ADP and SEP, and were scheduled for perfor- 
mance at the Category II test site following computer 
program installation and checkout, prior to FACI and 
the beginning of Category II testing. Successful com- 
pletion of FQT was planned as the primary testing basis 
for acceptance of the CPCEI. 

3. Test Procedures and Reports. Procedures and reports were required 

for both PQTs and FQTs, in conformance with BSD 262 and 263 (currently 
contained in Vol. II of AFSCM/AFLCM 310-1 as T-ld3 and T-118) . Test 
Procedures documents for PQTs were required for SPO review 30 days in 
advance of scheduled PQTs. In the case of FQTs, the Test Procedures 
document was required 90 days in advance, for SPO review and approval 
and to accomplish necessary changes. 

In summary, the Category I test program applied to BUIC III CPCEIs , other than 
utility, covered the full range of documentation and formal testing procedures 
which have since become incorporated in general guidance for 375-series appli- 
cations to computer programs. The novel elements, as compared with previous 
SDC/A16M SPO experience, were the preliminary and formal qualification tests 
and their associated test plans, procedures and reports. 

D. PROGRAM MILESTONES 

1. General 

A summary of major events as they actually occurred during the program is pre- 
sented graphically in Figure 5-1, which portrays significant dates relating to 
specifications, design reviews and FACI, test plans, and tests, principally 
for the three computer program contract end items (CPCEIs) in question. Dates 
on which the system specification was issued and the Category II testing was 
conducted are also included, however, for orientation to the system program as 
a whole. 

To distinguish among the three CPCEIs, different symbols are used for the Air 
Defense Program (ADP), the System Exercise Program (SEP), and the Utility Com- 
puter Program (UCP) ; these are triangles, circles, and squares, respectively. 
Symbols are drawn as solids for actual past events, and in outline only for 
scheduled future events which had not occurred at the time the chart was 
drawn (February 1969). As stated, the chart represents the timing of actual 
occurrence, prior to February 1969. It is not based on the initial program 
schedule, nor on periodic revisions which occurred as the program progressed. 
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Some of the events are depicted in the figure in a somewhat oversimplified 
manner, in the interests of avoiding clutter. However, more precise data, 
together with some amplification of their meaning in terms of related events 
and circumstances, are provided in the narrative below. In this section, 
emphasis is placed on descriptive and factual data regarding the program events. 
Discussions of problem areas and evaluative comments will be reserved for the 
following chapter. 

2. System Specification 

The system specification for BUIC III was initially issued as a formal document, 
designated SS-ES416M-65-1A, on 1 June 1965, It had been developed during the 
first six months of 1965 by the MITRE Corporation, principally, but with signi- 
ficant inputs from the two associate contractors, SDC and Burroughs Corporation. 

A major revision, designated SS-ES416M-65-1B, was issued exactly one year 
later, on 1 June 1966. The revision contained significant changes and expan- 
sions resulting from contractor efforts in developing Part I specifications 
for the system contract end items. 

These two events are depicted by star symbols in the top row of Figure 5-1. 

3. Part I Specifications 

The dates of Part I specifications which are shown in Figure 5-1 for the three 
computer program CEIs represent, in general, the principal dates of basic 
issue — i.e., of the first issues in accepted forms which constituted initial 
baselines for the items. In all cases, basic issues were preceded by one or 
more drafts for SPO review and approval. Hence, except for formally pro- 
cessed subsequent changes, the dates attempt to represent times at which the 
Part I specification developments were completed. Certain deviations and 
other amplifying information are noted below, separately for each of the 
CPCEIs. 

AIR DEFENSE PROGFAK (ADP) 

Because of its complexity and bulk (in excess of 2000 pages, total) the ADP 
Part I specification was structured into a set of 11 volumes, corresponding 
generally to a breakdown of the ADP into major functional groupings. This 
multi-volume structure was also adopted for roost of the other principal com- 
puter program documents, as will be explained subsequently for each. 

The date indicated in Figure 5-1 actually represents completion of only the 
first 10 of these volumes; the 11th, "Message Formats", was delayed for an 
additional six months by the need to resolve interfaces with other systems 
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(SAGE, BUIC II). A list of actual dates for all volumes, together with their 
numbers, titles, and functional elements covered under each, is presented in 
Table 5-2. For convenience, the table combines the Part I specification data 
with similar data relating to Category I Test Plans, which will be discussed 
in a later subsection. 

SYSTEM EXERCISE PROGFAM (SEP) 

The Part I specification for SEP was written in four volumes. The first two, 
"General" and "BUIC Exercise Preparation System (HEPS)," were issued on 21 
January 1966 and the second two, "BUIC Analysis and Reduction System (BARS)" 
and a "Classified Supplement," were issued on 1 September' 1966. Table 5-3 
lists these volumes, their titles, the functional elements comprising indiv- 
idual chapters under each, and the dates of basic issue. As in the case of 
ADP above, the table also includes Category I Test Plan data which will be 
discussed separately . 

The widely-separated dates of their Part I specification volumes reflects, in 
part, the fact that BEPS and BARS were relatively independent part of the SEP. 
A similar separation with respect to major milestones tended to be character- 
istic throughout the program. However, the late initial dates of BARS Part I 
specifications were largely a result of new requirements associated with the 
System Test Reduction Processor, the Subsystem Test Processor, and modification 
to the Test Data Analysis Processor. The new requirements, which were introduced 
shortly prior to the original completion date, added about 40% to the size and 
occasioned a realignment of contractor manpower and schedule. 

UTILITY COMPUTER PROGRAM (UCP) 

The UCP Part I specification consisted of four volumes, all of which were 
issued on 10 January 1966 as indicated in Figure 5-1, three weeks earlier 
than the ADP and SEP specifications. The left-hand column of Table 5-4 shows 
the volume numbers and titles of the UCP Part I specification. The second 
column in the table lists the functional elements which comprise individual 
chapters in the various volumes. Note, as indicated in Table 5-4, that one 
function, "Timing," was added to Volume 3 on 5 November 1968. This addition 
was the result of ECP 120-2, "Addition of a Timing Program to ADP/UCP," which 
provided for a timing tool to facilitate estimating the operating time of ADP 
programs or portions thereof. The third column in the table shows the dates 
of issue of the UCP volumes and the Timing function document. The last two 
columns contain data pertaining to Category I Test Plans, which are discussed 
in the following section. 

4. Category I Test Plan 

The Category I Test Plans for BUIC III CPCEIs, like the Part I specifications, 
were also structured into volumes, corresponding roughly to functional elements 
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Table 5-2- Volume Structure of Part I Specification and Category I 
Test Plan for the Air Defense Program (ADP) 



Part I Specification 


Functional Elements 


Dates 


Cat, 


I Plan 


Vol. 


Title 


Vol, 


Dates 


1 


General 




31 Jan 66 


1 


1 May 66 


2 


Air Surveillance 


Radar Inputs 

Active Tracking 

Passive Tracking 

Height 

Positive Target Control 

Identification 


31 Jan 66 


2 . 
3 

4 
5 
6 

7 


1 May 66 
1 May 66 
28 Jan 6^ 
21 Jan 66 
1 May 66 
1 May 66 


3 


Weapons 


Weapons Assignment 

and Commitment 

Weapons Direction/ 

Communication 

Air Defense Artillery 


31 Jan 66 


8 
9 
10 


25 Jan 66 
25 Jan 66 
25 Jan 66 


4 


Information 
Transfer and 
Manual Inputs 


Information Transfer 
Manual Inputs 


31 Jan 66 


11 
12 


24 Jan 66 
1 May 66 


5 


Startover, 
Control, Record- 
ing, and Real- 
Time Simulation 


Startover 

Control 

Recording 

Real-Time Simulation 


31 Jan 66 


13 
14 
15 
16 


1 May 66 
1 May 66 
1 May 66 
1 May 66 


6 


Adaptation 




31 Jan 66 






7 


Variable Display 
Equipment 




31 Jan 66 






8 


Displays 


Display Specification 
Situation Displays 
Tabular Displays 
Abbreviations and 
Definitions 


31 Jan 66 


17 


1 May 66 


9 


Switch Actions 




31 Jan 66 






10 


Classified Suppleme 


nt 


31 Jan 66 






11 


Message Formats 




1 Aug 66 
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Table 5^3. Volume Structure of Part I Specification and Category I 
Test Plan for the System Exercise Program (SEP) 



Part 


I Specification 


Functional Elements 


Dates 


Cat. 


I Plan 


Vol. 


Title 


Vol. 


Dates 


1 


General 


Control 

Data Base Maintenance 

Octal Correction 


31 Jan. 66 


1 




2 


Exercise Pre- 
paration (BEPS) 


Exercise Tape 
Generation 
Exercise Tape 
Description 
Exercise Tape 
Modification 


31 Jan. 66 


1 
1 
1 
1 


1 May 66 


3 


Analysis and 
Reduction 
System (BARS) 




1 Sept. 66 










Exercise Processor 
Operational Processor 
Test Data Analysis 




2 
3 
4 


15 Nov. 66 
15 Nov. 66 
15 Nov. 66 






Processor 

System Test Reduction 




5 


15 Nov. 66 






Processor 
Subsystem Test Pro- 




6 


15 Nov. 66 






cessor 








4 


Classified 
Supplement 




1 Sept. 66 
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Table 5-4. Volume Structure of Part I Specification and Category I 
Test Plan for the Utility Computer Program (UCP) 



Part I Specification 
Vol. Title 



Functional Elements 



Dates 



Cat. I Plan 



Vol. Dates 



1 General 

2 Assembly Analysis 



Facility System 



General Utility 



10 Jan. 66 



10 Jan. 66 



Machine Language 
Assembler 
JOVIAL Compiler 
Assemble Compool 
Assemble Geography 
Binary Data Insertion 
Tape Load 
Set/Use 

Indirect Address and 
BAR Table Reference 
Adaptation Calculation 
Parameter Test Tool 



Facility Control 

Pre-Recording 

Dynamite 

Symbolic Relative 

Corrector 

Timing 



Tape File Maintenance 

Binary Read 

Load Octals on Tape 

Tag Reference 

Symbolic Corrector 

Loader 

Input/Output 

Computer Utility and 

Support System Executive 

Dump Function 



1 May 66 



1 May 66 



10 Jan. 66 



(added 5 
Nov. 68) 

10 Jan. 66 
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of the computer program. In general, basic issues of test plan volumes were 
preceded by drafts for SPO review and approval. While they are not directly 
subject to configuration management, the plans were also maintained on a 
continuing basis during Acquisition to reflect approved changes to the Part I 
specifications. Accordingly, they were frequently updated by means of change 
pages or revision, following their dates of basic issue. 

In Figure 5-1, symbols are located at points which represent the basic issue 
dates for groups of volumes. Additional explanation is provided below for 
each of the three computer program items. 

AIR DEFENSE PROGRAM (ADP) 

The Category I Test Plan for ADP was written in a total of 17 volumes. One 
of these (Vol. 1, General) emphasized planning of formal qualification (FQT) 
for the CPCEI as a whole. The other 16 were devoted individually to detailed 
planning of PQTs for the computer program functional elements identified in 
various volumes of the Part I specification, primarily, but also included 
planning pertinent to FQT. 

Symbols shown in Figure 5-1 are located at two different dates for ADP, 
indicating that 6 volumes were issued in January and the remaining 11 volumes 
in May of 1966. The volume numbers, titles (corresponding to the Part I 
specification titles shown), and dates of issue are listed in the preceding 
Table 5-2. It may be noted that the table shows an absence of test plan 
volumes corresponding to switch actions and message formats. These were 
necessarily included as essential aspects in the testing of other functional 
elements. 

SYSTEM EXERCISE PROGRAM (SEP) 

The two points shown in Figure 5-1 for the SEP Category I Test Plan also 
indicate two separate dates of issue for the test plan volumes, in May and 
November 1966. The volumes, dates, and correspondence of test plan breakdown 
with elements of the Part I specification are identified in the preceding 
Table 5-3. The discrepancy of 6 1/2 months in publication dates reflects, 
basically, the lag in BARS portions of SEP which had been initiated by the 
earlier delay in completing BARS volumes of the Part I specification, 

UTILITY COMPUTER PROGRAM (UCP) 

An early decision had been reached that utility computer programs, except for 
two designated elements, would be qualified through their use in developing the 
ADP and SEP. As a result, the test plan for UCP was confined to one volume 
each for the two designated elements, the JOVIAL Compiler and BUIC Adaptation 
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Calculation (BAG). These are listed In the preceding Table 5-4. In Figure 
5-1, the symbol represents the date on which both volumes were published. 

5. Preliminary Design Reviews (PDRs) 

PDRs for the three CPCEIs were held on the following dates: for ADP, on 14-15 
February 1966; for parts of SEP, on 15-16 February 1966; for UCP on 16 
February 1966; and for remaining parts of SEP, on 23 August 1966. The two 
Increments of PDR for SEP basically followed the discrepancy between BEPS and 
BARS which had been generated earlier by the delay In the BARS Part I specifi- 
cation. However, the February PDR, for BEPS, also covered the BARS Test Data 
Analysis Processor. The later Increment was devoted to the BARS Exercise 
Processor, Operational Processor, System Test Reduction Processor, and Sub- 
system Test Processor. 

The review in each case was concerned with preliminary design of the CPCEI as 
a whole, with respect to levels of design information which would subsequently 
comprise a general volume (Vol. 1) of each Part II specification. These 
Included attention to Interfaces, allocation of functions to computer program 
components (CPCs) , control and sequencing of CPC operations, and storage 
allocations. The preliminary design information to be reviewed was documented 
and delivered to the SPO for advance examination, prior to each PDR. Results 
of the reviews were subsequently Incorporated in PDR Minutes, which were pre- 
pared by the contractor and approved by the SPO. 

6. Critical Design Reviews (CDR) 

The basic approach adopted in BUIC III was to conduct CDR Incrementally in 
association with PQTs.* For this reason, the chart in Figure 5-1 shows a 
single row of combined PQT/CDR events for each CPCEI. The precise dates and 
numbers are discussed further under PQTs, in the next subsection. 

However, in the case of SEP, some elements underwent CDRs which were not 
associated with PQTs. These were cases in which combined PQT/CDRs had been 
planned initially, but for which PQT requirements were subsequently waived. 



* BSD Exhibit EST-1 (Sec. H; Exh. XIV) defines CDR for CPCEIs as basically a 
review at trie level of logical design of individual CPCs, prior to coding 
and testing. However, it also permits "flexible application" in the case 
of CPCEIs whose development follows an Incremental pattern during Acquisition, 
suggesting that the CDR may be held in corresponding Increments and may be 
combined with PQTs. 
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The SEP portions affected, and the dates of their CDRs, are listed below: 



BEPS 

Exercise Tape Generator 
Exercise Tape Description 
Exercise Tape Modification 

BARS 

Test Data Analysis Processor 



15 January 1968 



1 February 1967 



Where PQTs were held, material reviewed at CDR was the available Part II 
specification material for the elements undergoing PQT. Hence, it was at 
the level, in essence, of completed design. It is reported that items 
specifically reviewed included, for each given CPC or combination of CPCs, 
the following: CPC name(s), functions, size, and interfaces; emphasis was 
placed on CPC sizes, noting comparisons with BUIC II and deviations from 
PDR estimates. It has also been reported that questions of timing were 
raised with increasing interest as the program progressed towards later phases. 

7. Preliminary Qualification Tests (PQTs) 

As indicated in Figure 5-1, PQTs occurred in large numbers for the ADP and SEP 
CPCEIs. Initial planning of PQTs was based on the philosophy that preliminary 
qualification testing was required for each and every element of the computer 
program. Accordingly, they were so defined and scheduled in the ADP and SEP 
Category I Test Plans. 

As was noted in the preceding subsection, each PQT was initially planned to 
occur in combination with an incremental CDR of the element (s) being PQT'd. 
Detailed data relating to these events as they actually occurred are set forth 
below, separately for each of the CPCEIs. 

AIR DEFENSE PROGRAM (ADP) 

The record shows that a total of 35 PQTs were held for the ADP, of which 32 
were combined with CDRs. Listed below are the dates of occurrence, and 
following each the number of PQT/CDRs held on that date. The last three dates 
listed are the occasions on which PQTs were not accompanied by CDRs. 



19 September 1966 (2) 
1 November 1966 
7 March 1967 (4) 

25 July 1967 
25-26 July 1967 



26 July 1967 (2) 

22 August 1967 

23 August 1967 

24 August 1967 

12 September 1967 



13 September 1967 (2) 

14 September 1967 (3) 

10 October 1967 

11 October 1967 (3) 
21 November 1967 (3) 
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5 December 1967 (3) 
15 December 1967 (2) 



16 January 1968 
30 January 1968 



26 February 1968 



In some cases, a single PQT/CDR was held for an entire CPC (see Table 5-5). 
For selected components of ADP, CDRs were conducted in increments as the 
design and testing of pieces of a CPC progressed. In the case of incremental 
CDRs, areas of a computer program which did not have a PQT at the first CDR 
for that computer program had a PQT at a subsequent CDR. Thus some CPCs 
underwent as many as three CDRs. Table 5-5 shows the number of CDRs held 
for each CPC in the ADP CEI and in the order they occurred. The Minutes for 
each CDR list the subroutines and "functional areas" of the CPC which, at a 
particular time, did or did not have a PQT and which did or did not undergo a 
CDR. 



Table 5-5. PQTs/CDRs for ADP CPCs 







No. of 






No. of 






No. of 






CDRs 






CDRs 






CDRs 


1. 


MIN 


- 2 


11. 


TRK - 


1 


21. 


SIG - 


1 


2. 


BIT 


- 1 


12. 


RAC - 


3 


22. 


TAR - 


1 


3. 


BOB 


- 3 


13. 


MID - 


1 


23. 


SET - 





4. 


AAD 


3 


14. 


KAW - 


1 


24. 


CIO - 


* 


5. 


BIP 


- 2 


15. 


RAP - 


2 


25. 


KAS - 


* 


6. 


OCS 


1 


16. 


COP - 


1 


26. 


SRT - 


* 


7. 


REC 


1 


17. 


SID - 


1 


27. 


ART - 


* 


8. 


BID 


1 


18. 


TAC - 


1 


28. 


SQR - 


* 


9. 


TOP 


1 


19. 


TAG - 


1 


29. 


TNC - 


* 


10. 


BIO 


- 2 


20. 


BOK - 


1 


30. 


WND - 


* 



It should be noted, as shown in Table 5-5, that one CPC, SET, in the ADP CPCEI 
did not have a CDR, and seven CPCs (CIO, KAS, SRT, ATR, SQR, TNC, and WND) 



* These CPCs underwent CDRs as parts of other CPCs, 
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Table 5-^6. Numbers of ADP PQT Procedures and Reports by Functions 
and CPCs 



Function or CPCs 


Procedures 


Renorts 


1. 


Radar Inputs 


2 


3 


2. 


Active Tracking 


1 


3 


3. 


Passive Tracking 


2 


4 


4. 


Height 


2 


3 


5. 


Positive Target Control 


3 


3 


6. 


Identification 


3 


5 


7. 


BOMARC Guidance 


4 


4 


8. 


Manned Interceptor Guidance 


4 


4 


9. 


BOMARC Prelaunch 


2 


2 


10. 


TOLL 


1 


1 


11. 


Air Defense Artillery 


2 


6 


12. 


Information Transfer 


6 


6 


13. 


Manual Inputs 


2 


2 


14. 


Startover 


1 





15. 


Control 


1 


1 


16. 


Recording 


1 


1 


17. 


Simulation Information and Tape Read 


3 


3 


18. 


Tabulation and Situation Displays 


4 


4 


19. 


Weapons Assignment and Commitment 


4 


4 


20. 


Weapons Illegal Switch Action 


1 


2 


21. 


Surveillance Illegal Switch Action 


1 


2 


22. 


Illegal Switch Actions 


2 







Total 


.s 52 


63 
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underwent a CDR and testing as parts of the CPCs utilizing them. A total of 
32 CDRs were conducted. Those PQTs conducted in January, February, and March 
1968 for CPCs which had previously undergone CDRs did not have CDRs a second 
time. 

Although participants report that the CDR discussions did result in a number 
of actions, including design changes, essentially no actions are recorded in 
the CDR minutes. This circumstance is apparently a function of the fact that 
minutes were prepared only once, after the 18-month series of CDR increments 
was completed. One can guess that the record of actual events would have been 
better had the minutes also been prepared in corresponding increments. 

For the ADP CPCEI, a total of 53 PQT Procedures documents and 63 Test Reports 
were published. The General PQT Procedures document was issued on 23 June 
1966, with the balance issued during the remainder of 1966 and throughout 1967 
and 1968. The earliest Test Reports were issued in October and November 1966, 
with the balance of the reports widely distributed throughout 1967 and the 
months of January, February, and March 1968. 

The large number of Procedures documents (52) is due to the fact that for some 
functions several PQT procedures were prepared, as shown in Table 5-6. The 
table shows the number of Procedures documents issued for each function 
identified in the Part I specification. It should be noted that the Pro- 
cedures and Test Reports do not match the functions in the Part I specifi- 
cation on a one-to-one basis in many cases, but are also related to CPCs. 
The table also shows the number of Test Reports (63) for the same functions or 
CPCs. In some cases there are more Reports than Procedures as a result of 
test re-runs. 

SYSTEM EXERCISE PROGRAM (SEP) 

According to the formal CDR Minutes for the BARS portion of SEP, PQTs/CDRs 
were conducted on the dates listed below. The number in parentheses shows 
the number of CPCs for which PQTs/CDRs were held on that date. 

System Test Reduction Processor 

11 December 1967 (3) 16 January 1968 (3) 

12 December 1967 (4) 17 January 1968 

13 December 1967 13 March 1968 
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Operational Processor 

11 December 1967 (5) 
14 December 1967 
18 January 1968 (2) 



13 March 1968 

14 March 1968 



Exercise Processor 



14 March 1968 



Various materials were made available for use as necessary in accomplishing 
the CDR for each CPC. These included the Part I specification, a list of ADP 
items and tables used, and drafts of the Part II specification including 
programs listings, and users manuals. The list of items and tables used for 
each computer program or functional area was usually part of the PQT Pro- 
cedures document. 

For the SEP CPCEI, a total of 30 Procedures documents and 27 Test Reports were 
published. The General PQT Procedures document was issued on 23 November 1966, 
Others were issued over a six-month period from October 1967 to March 1968. 
The Test Reports were published in June, September, and December 1967, and in 
January and March 1968. 

UTILITY COMPUTER PROGRAM (UCP) 

For the UCP CPCEI, Table 5-7 shows the dates for the conduct of PQTs/CDRs, and 
the dates of issue of the PQT Procedures and Test Reports.* The SPO did not 
send representatives to the CDRs conducted in conjunction with the PQTs on 
1 August and 30 July 1967. 

Table 5-7. Dates of UCP PQTs and Basic Issues of Procedures and Reports 



Functions 


Procedures 


PQTs/CDRs 


Reports 


JOVIAL 


1 July 1967 


1 Aug. 1967 


1 Sept 1967 


Adaptation Calculation (BAG) 


30 June 1967 


30 July 1967 


30 Aug. 1967 



* Out of a total of 67 UCP CPCs, only 8 underwent a PQT, 
for the Compiler and 1 CPC for BAC. 



These include 7 CPCs 
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8. Formal Qualification Test (FQTs) 

As indicated in Figure 5-1, FQTs were performed only for the ADP and SEP con- 
tract end items. In both cases, FQTs were conducted in a number of separate 
parts, as identified in the list below. A separate FQT Procedures document 
and Test Report were written for each part. 

AIR DEFENSE PROGRAM (ADP) 

Simulation Mode Test 
Interface Test 
Live Mode Test 
System Load Test 

SYSTEM EXERCISE PROGRAM (SEP) 

BEPS: Exercise Preparation System 

BARS: Exercise Processor 

Operational Processor 

System Test Reduction Processor 

Subsystem Test Processor 

The initial FQTs for all parts were conducted at the Category II site (BUIC 
Evaluation Facility (BEF) , located at Hanscom Field, Massachusetts) during the 
period 2 April through 16 April 1968, with the exception that the Subsystem 
Test Processor had been FQT'd at the contractor's plant on 12-14 September 
1967. 

Because of a major deficiency relating to cycle time of the ADP, which had 
been known to exist at an earlier date but had not been overcome by the time 
of the April FQT, a complete re-run of the ADP FQT was scheduled and performed 
in December 1968 to verify corrections which had been accomplished by that 
time. The symbol shown for that month for ADP in Figure 5-1 represents this 
re-run. 

Scheduled additional FQTs at future dates may also be noted in the chart. 
These represent (a) a major new version of ADP, with corresponding changes 
reflected in SEP components, to incorporate new operational capabilities 
beyond those for which requirements had been established earlier in Acquisition 
and (b) a scheduled updating of CPCEIs associated with turnover of BUIC III 
to the user. 
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9. Part II Specification 

The Part II specification for each of the three CPCEIs was subdivided into 
an extensive set of volumes. Each had a first, General volume describing 
the design and structure of the CPCEI as a whole. Additional volumes were 
devoted to individual computer program components (CPCs), for the most part, 
although a few dealt with data or other special items such as the Compool 
Description, Constants, Adaptation, and Library. 

The 30 CPCs of the ADP are listed in Table 5-8 as an example. The ADP 
specification contained 38 volumes in all, while there were 141 volumes for 
SEP and 66 for UCP. As a complete set, the Part II specifications for the 
three CPCEIs contain approximately 40,000 pages of computer program design 
documentation. Drafts of some volumes of the Part II specifications were 
issued as early as February 1967. Many of the drafts were revised and 
reissued as many as 3 or 4 times prior to the date of 15 April 1968, when all 
volumes for the three CPCEIs were delivered for review prior to FACI. The 
15 April date, as shown in Figure 5-1, therefore represents the first 
"completion" date for the Part II specifications. However, they were still 
pre-FACI drafts at that time, rather than basic issues. 

Potential problems of Part II specification timing in relation to FQT, FACI, 
and Category II testing had been recognized early in the program. In reaching 
a solution to these problems, account was also taken of the additional factors 
introduced by the on-going processing of changes. The solution adopted is 
outlined briefly below. 

An artificial "freeze" of the computer program was instituted to reflect com- 
pletion of the Part II specification drafts, shortly prior to their issue 
date of 15 April. This configuration represented Version 1 of each CPCEI, 
which was the version to be FACI'd. Subsequently, the drafts would be 
re-published as basic issues of the Part II specification — constituting the 
initial baselines, but would contain no updating or corrections other than 
those resulting from SPG comments at FACI. As time passed and the intervening 
events occurred, these basic issues were eventually completed (for ADP) in 
February 1969. Except for a few volumes, they had continued to reflect 
Version 1 in its frozen configuration. 

However, changes continued to occur, in actuality, throughout that period. 
ECPs had been in process at the time of the freeze, and more changes resulted 
from FQT, Category II testing, and other causes. One major set of changes 
was introduced to overcome the cycle time problem (see next chapter) in ADP; 
and Version 7 .of ADP underwent a second FQT to test those changes in December 
1968. All changes through Version 7 were formally processed and issued, 
simultaneously with the basic issue, as product configuration baseline changes 
in February 1969. Thus, in effect, the initial product configuration baseline 
was established with the issuance of pre-FACI drafts. 
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Beyond the date of pre-FACI drafts, additional events depicted in Figure 5-1 
represent: (a) completion of basic issues for SEP and UCP late in 1968; (b) 
the major updating of ADP in February 1969; and (c) a projected complete 
revision of all Part II specifications at the time of turnover to the user 
in April 1970. 

10. First Article Configuration Inspection (FACI) 

The first FACI events shown in Figure 5-1 represent the FACI which was 
accomplished on 20-24 May 1968, approximately a month following delivery of 

pre-FACI drafts of the Part II specifications, for all three CPCEIs. One 
element of SEP, the Subsystem Test Processor, which had previously undergone 
an in-plant FQT, had also passed an in-plant FACI in September 1967. FACI 
was completed at that time for all other elements of the CPCEIs, except for 
5 volumes of the ADP specification dealing with CPCs which were undergoing 
known major changes associated with the cycle time problem. As a result, 
these were reserved for a subsequent FACI "increment" which was held following 
delivery of Version 7 (the Timing Version) of ADP in February 1969. 

11. Category II Testing 

The Category II tests for BUIC III were conducted at the BUIC Evaluation 
Facility (BEF), located at Hanscom Field, Massachusetts, during the 4-month 
period of 15 April to 15 August 1968. The BEF, which was created as an ESD/ 
MITRE test facility at Hanscom, was configured as an operational BUIC III site. 
Following Category II tests, it was closed on. 16 August, but was later 
re-opened for additional testing associated with BUIC III. This additional 
testing included the repetition, in December, of the Timing Version FQT for 
the ADP, except for the Live Mode Test which was conducted at operational 
site Z-10, North Truro, Massachusetts. 
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CHAPTER VI 
DISCUSSION AND EVALUATION 



A. GENERAL 

1. Qualifying Factors 

BUIC III has been referred to as a "test bed," in which the novel elements 
described in Section C of the preceding chapter are thought of as the experi- 
mental variables whose effects on the system program are being identified and 
evaluated. From that point of view, it is one purpose of this report to 
assess the degrees of success and failure associated with the BUIC III applica- 
tions of those management techniques, and to evaluate them accordingly. How- 
ever, it should be recognized that the mere presence of the requirements did 
not automatically insure that they all became effective influences in the 
program. In fact, there were circumstances and constraints operating through- 
out to preclude simple interpretations in terms of success or failure as 
judged against the stated requirements. 

The very novelty of the requirements meant that many of their complex implica- 
tions for planning and implementation were not readily understood and appreciated. 
Supplemental guidance, interpretations, and explanations to accompany the 
formal statements of the requirements were minimal or nonexistent; and there 
was a corresponding absence of experience upon which to base realistic planning. 
The detailed implementing solutiorts were worked out as the program progressed, 
with the net result that some of the requirements were actually implemented in 
much the way that might be expected; some tasks were relatively unaffected; 
and others managed to reflect varying degrees of compromise with tradition, or 
to take unique and unexpected forms. 

The Category I Test Plans were outstanding products of that initial inexperience 
and confusion, as the first to be contended with among the many novel elements 
associated with Category I testing. The writers of the plans attempted to 
define the spectrum of requirements and planning factors for the computer pro- 
gram CEIs in fine detail. The Part I specification Sections 4 were written 
from the plans, rather than the reverse. Functions of the plan were not 
distinguished from those of PQT and FQT Procedures documents to follow; 
planning centered around functional elements of the Part I specifications was 
not clearly relatable to structural components of the computer programs; and 
the planning often tried to encompass the entire gamut of computer program 
test activities without regard to distinctions between internal development and 
formal testing aspects. The resulting bulk, detail, and misconceptions 
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incorporated in these plans (for ADP and SEP) set the stage for many troubles 
which ensued in carrying out the Category I testing efforts (see Section E 
below) . 

The fact that BUIC III was a modification of BUIC II, rather than a new system, 
was a further factor which interfered with adopting some of the new techniques 
in their intended form. It was a general requirement in the program to utilize 
existing BUIC II content and materials as much as possible. Conceptual phase 
studies were limited, for the most part, to the new features; and there was 
no formal Definition phase. Additionally, it should be noted that the areas in 
which the new techniques were introduced — namely, configuration management, 
data management and testing — did not include system engineering management 
techniques drawn from AFSCM 375-5, with the relatively minor exception of design 
reviews. Yet, within the general framework of 375-series concepts, system eng- 
ineering is recognized as the fundamental process upon which other techniques 
depend for their substantive content and execution. 

At the time this report is written, various problems associated with adapting 
the 375-5 principles to data processing elements of systems still remain to be 
explored and resolved. In BUIC III, the system engineering accomplished was 
necessarily of an ad hoc and traditional nature — which meant that it was a 
combination of systematic analysis, trial-and-error explorations, and previous 
experience, with the technical emphasis naturally devoted to areas which had 
proved to be important or troublesome in earlier systems. 

The problems encountered with cycle time of the operational computer program 
(see Section F below) can be largely attributed to the combined effects of 
these various factors — i.e., limited Conceptual phase, no Definition phase, 
dependence on BUIC II precedent, and associated absence of system engineering 
analyses which were sufficiently comprehensive to detect a potential timing 
problem at early points in the system life-cycle. In fact, the effects of these 
factors were noticeable throughout the Part I specifications. Organization, 
style, and content of the BUIC III Part I specifications were basically 
patterned after their "ops spec" predecessors, even though they carried the 
new titles and formats dictated by the Form 9. 

2. Program Overview 

Discussions of the specifications, testing, configuration control, and other 
individual topics are to be provided in subsequent sections of this chapter. 
Before discussing those topics individually, however, attention is directed 
briefly in this section to the series of events as a whole, based on their 
sequencing during the Acquisition phase as depicted in Figure 5-1 of the 
preceding chapter. 

Certain general impressions of the total program can be gained by examining 
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the actual BUIC III sequence of milestones in the light of expectations based 
on 375-series systems management concepts. The chart may be compared, in 
particular, with the model "road maps" contained in ESD TR 68-1 for the com- 
puter program system segment, as they were adapted from such sources as 
AFSCM 375-4 and 375-5 • 

The timing of events during the first year of Acquisition appears to fit fairly 
well the pattern one might expect from a system program having no Definition 
phase. That is, ignoring a few delays in particular parts of the items, (a) 
the System Specification initiated both the development of Part I specifi- 
cations and preliminary design of the CEIs, (b) the Category I Test Plans were 
closely related in time to the Part I specifications, (c) the PDRs were held 
at the completion of preliminary designs, and (d) the System Specification 
underwent a major update and expansion as a result of the Part I specification 
development efforts. 

One would not normally expect to see PQTs held on so many occasions as they 
were for both the ADP and SEP CEIs. In BUIC III, the time and effort expended 
on PQTs were not only greater than expected, but were also far more costly 
than the results justified. The problems generated by this abnormal (one hopes) 
PQT program are discussed at length in section E below. 

The timing of FQTs, at their first appearance immediately preceding the 
initiation of Category II tests, may be considered normal for operational com- 
puter programs, although it would be later than normal for equipment items. 
However, the continued appearance of FQTs over a time span of one and one-half 
years after the completion of Category II testing is a surprising circumstance 
for any type of contract end item. These continuations are identified as 
arising from two types of causes: (a) the necessity to re-run FQT as a result 
of deficiencies in meeting the original requirements; and (b) the need to verify 
follow-on versions of the original CEIs, containing changes to meet extensive 
new requirements. Again, the questions associated with this and other aspects 
of the Category I test program are discussed further in Section E below. 

The timing of Part II specifications and FACI occurred largely as planned in 
relation to the initial FQT and Category II test events, except for (a) a 
change package containing major revisions of the Part II specifications for 
ADP associated with the deficiency noted above, and (b) a delayed "increment" of 
FACI for ADP associated with the same deficiency- The chart also depicts a 
planned future revision and reissue of the specifications, occasioned by turn- 
over of the system to the using command. 

At the most general level, it seems worthy of some note that the events shown 
were actually incorporated in the program and accomplished. While they may 
now be taken for granted in the light of the past few years* experience, it 
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may be remembered that not one of the subsystem-level milestones listed in the 
chart had appeared in information processing system programs prior to BUIC III. 
Collectively, they reflect a significant trend towards increased visibility of 
elements in the computer programming process, in the direction of permitting 
SPOs to assume increasingly active roles in the management of computer program 
acquisitions. 



B. SPECIFICATIONS 

1. Part I Specifications 

The influence of some of the factors mentioned in the preceding section, 
affecting the degree to which formal requirements were actually implemented in 
BUIC III, was most noticeable in items produced at early points in the program. 
Thus, while a considerable effort was devoted at a later time to developing 
Part II specifications which would conform to the full intent of the Form 9, 
the Part I specifications retained the flavor and much of the content of 
their BUIC II predecessors, the operational specifications. 

The operational specifications had relied heavily on narrative description of 
system organization and operations. Much of their content was informational, 
rather than being written in the mandatory language of a "design-to" specifica- 
tion. In some ways they were as much reports of the analysis/definition pro- 
cess as they were specifications in a legal sense. For example, they contained 
"scenarios" of system data processing operations which often included human 
operator actions. While such scenarios are useful system engineering devices 
associated with the process of analyzing and identifying functional detail, 
and constitute levels of description which are readily understood by a user, 
they do not tend to provide unequivocal definitions of the computer program 
CEI requirements. 

In effect, the BUIC III Part I specifications were transitional documents. 
They contained new sections/subparagraphs conforming to the Form 9 in such 
areas as interfaces, inputs and outputs, data base, human performance, and 
quality assurance, although not to the extent of resolving a variety of 
questions regarding the ways in which some of those topics should be treated. 
The comments below indicate types of problems which have been encountered in 
the course of developing Part I specifications for other systems, e.g., SEEK 
DAWN, as well as in BUIC III. 

a. The level of information to be specified in the Part I is 
relatively well defined by the Form 9 (ESD 236) in some 
areas, e.g., input and output messages, but is subject to 
a wide range of interpretation in others. The inclusion 



72 



in any Part I specifications of many mathematical formulas 
and equations has been questioned specifically, where they 
might be regarded as being design solutions rather than 
performance requirements as such. The BUIC III Part I's 
contained many of these, most of them being solutions which 
were known as a result of BUIC II design or because BUIC III 
design was being carried out concurrently with the writing 
of the Part I specification. Many were necessarily incor- 
porated as design requirements, since they had been previously 
spelled out at that level in the System Specification, also 
because of known BUIC II design. Otherwise, and with 
reservations noted elsewhere, the level of detail represented 
in BUIC III is considered appropriate and necessary for 
adequate definition of CPCEI performance characteristics — 
that is, for the Part I specifications in their completed 
form. Generally, much of the required information would not 
be available for completing the specifications at that level 
by the end of a Contract Definition Phase effort. 

Information called for under paragraph 3.1.3, "Data Base 
Requirements," in the Form 9 is relatively clear as regards 
level of detail, but the types of data to.be specified have 
proved to be continuing matters of debate. The ADP specifi- 
cation in BUIC III confined its coverage to adaptation data, 
which are specifically called out in the Form 9. Many other 
types of data were specified, but under the various functional 
elements rather than under the data base paragraph. They in- 
cluded computational constants, weather data, radar inputs, 
weapons characteristics, and others, some representing data 
resulting at intermediate stages of processing. Different 
solutions have been proposed in various subsequent cases, 
ranging from the limited treatment exemplified by BUIC III 
to the extreme position, as in the SEEK DAWN/818, that the 
data base paragraph should encompass all items in the 
computer which contain data values as distinct from com- 
puter instructions. However, the merits of these different 
solutions seem to involve a variety of technical and manage- 
ment considerations which need to be further explored. The 
general problem is apparently more complicated in the real- 
time air defense context than it would be for some other 
types of systems, e.g., 473L, in which the computer programs 
function principally to retrieve information from a massive 
data base. 

Questions have also been raised as to whether the Part T 
specification should be structured to permit, or require. 
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the computer program to be designed with components (CPCs) 
having a one-to-one correspondence with the Part I func- 
tional elements. Currently, one-to-one correspondence is 
not a requirement, for the reason that it would likely in 
many cases restrict the freedom to strive for maximum 
efficiencies in computer program design. However, the 
absence of correspondence also creates some problems. In 
BUIC III, the most noticeable difficulties were in the 
testing area. In cases where CPCs, or groups of CPCs, 
were not directly identified with given elements of the 
Part I specification, complications were met in the planning, 
technical preparation, and conduct of performance testing. 
In writing the Part I specification for SEEK DAWN/818, a 
related problem was encountered in specifying inputs and 
outputs for the functional elements, as required under 3.1.2.1. 
Inputs and outputs of the CPCEI as a whole are necessarily 
the same at the Part I and II levels, but the internal inputs 
and outputs are another matter. The task of defining all 
of those is a formidable one, in itself, and it can prove to 
be of dubious value where the functional elements undergo 
rearrangements in the course of computer program design. 
While a matrix can be constructed (as it normally is) to 
depict the allocation of functional elements to CPCs, a 
similar matrix relating internal inputs and outputs as 
between the Part I and Part II. levels becomes far more 
complex. To illustrate, using the BUIC III ADP numbers 
of 15 functional elements and 30 CPCs, there can be 105* 
sets of input-output relations among functional elements 
to keep track of. To the degree that re-structuring 
occurs, many of these would not be included within the 
435 sets (i.e., maximum possible) of inter-CPC inputs 
and outputs, since some number of functional element 
inputs/outputs would be contained inside CPCs. Hence, 
many are lost, many are scrambled, and many more new inputs 
and outputs appear which are not identified as such at the 
Part I level. In the real case, considerable complications 
are added by such factors as variability of inputs and 
interactions among components. The net result is to pose 
problems of design and development which may not be 
clearly soluble, as well as to complicate the testing of 
individual functional elements. Testing of the integrated 



* That is, for K elements in general, there can be as many as K (K-l)/2 non- 
duplicated sets of input/output interface relations. 
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CPCEI should not be affected, however, since the identities 
of external inputs and outputs are presumably maintained in- 
tact. 

d. The proper content to include under the human performance 
paragraph 3,1.4 has also been a subject of frequent dis- 
cussion. A "model" treatment of this topic has not yet 
appeared among the Part I specifications for BUIC III 
and the Southeast Asia systems. A deliberate attempt which 
was made in SEEK DAWN/818 did result in a few examples of 
appropriate material. The examples were necessarily limited 
and to some degree artificial, however, as a result of having 
to be formulated after the designs of both computer program 
and equipment operating stations were completed. It has 
been observed that the proper content of the human performance 
paragraph in a computer program Part I specification is not 
directly analogous to what it would be for equipment items. 
In the case of equipment, it consists largely of requirements, 
e.g., as set forth in MIL-STD-803, which govern subsequent 
engineering design with respect to workspace layouts, noise 
and illumination factors, and display/control elements. For 
computer programs, the relevant display, control, and func- 
tional requirements associated with efficient man-machine de- 
sign must be incorporated into the detail contained within 
the Part I specification itself; the computer programmer has 
no freedom later to affect those solutions. Thus, if the 
necessary human engineering design is to be accomplished 
at all, it must be done as an integral part of the system 
engineering effort leading to the Part I specification. As 
yet, in the systems referred to, there have been no structured 
Conceptual and Definition phases which might have provided 
opportunities for that kind of systematic development; 
and much of the process, which one might expect to be 
analogous to the AFSCM 375-5 model except for significant 
modifications appropriate to the data processing context, 
remains to be explored in a realistic setting. The new 
system programs which do have Definition phases should 
provide additional experience in this area. 

2. Part II Specifications 

Requisite information to be contained in the Part II specifications for the 
BUIC III CPCEIs was identified by the Form 9, ESD 237, together with back-up 
sheets. SDC, early in the development effort of the BUIC III computer programs 
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prepared and submitted to the SPO an example Part II volume portraying the level 
of information that was planned. The example served as the basis for arriving 
at mutual agreement with regard to the volume structure and level of informa- 
tion content of the Part lis. The agreements, however, did not prevent 
difficulties which subsequently arose. The following discussion treats the 
significant aspects of the Part lis based on SDC's experience. The discussion 
concentrates on the Part II volumes for ADP since differences peculiar to the 
Part lis for SEP and UCP are not regarded as being of sufficient significance 
to warrant explicit treatment. 

The Form 9 permits structuring the Part II specifications into multiple volumes. 
This was done for BUIC III and done quite satisfactorily. Volumes 1 through 8 
contain information essential to programming tasks performed following initial 
program development (e.g., effecting program changes, adapting the computer 
program to unique environments) . This phase is normally referred to as the 
program maintenance phase. No unusual difficulty has been noted in reference 
to these volumes. 

By contrast, the content of the computer program component-oriented Part II 
volumes has provoked some discussion and concern. Most significant are the 
following: 

a. Description 

b. Flow Charts 

c. Interface 

d. Limitations 

e. CPC Data Organization 

In this context, it should be noted that the format and content of these CPC- 
oriented Part II volumes are intended to satisfy the requirements established 
by paragraph 3.2.1 of the Form 9. The requirements demand a brief abstract 
of the functions of the CPC, the languages in which it is written, and its 
major interfaces; additionally, the requirement states that the CPC shall be 
described in detail in subparagraphs. It is this last requirement that bears 
discussion with regard to Chapter 1, Description, and Chapter 2, Flow Chart, 
of the CPC Part lis which correspond to subparagraphs 3.2.1.1 and 3.2.1.2 of 
the Form 9. 

An examination of Part II volumes reveals that the presentation of descriptive 
detail contained in Chapter 1, Description, varies from one volume to another. 
This is to be expected considering the varying complexity of the CPCs and the 
varying ability of the programmers to describe the CPCs in narrative prose. 
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However, the point to be made here is simply that the value of much of the 
detail was sufficiently diminished by poor writing that the presumed purpose 
of the description was not satisfied. That is, that the primary subsequent 
function of the description is to convey an understanding of the operations 
performed by the subject CPC to a programmer who has responsibility for 
effecting changes and corrections to the CPC in the maintenance phase. 

A similar comment applies to the level of detail presented in Chapter 2, Flow 
Chart. It is suggested that this information should also be at a level of 
detail describing the operations performed by the CPC, the sequence in which 
they are performed, and graphically portray the CPC in sufficient detail to 
enable a programmer to ascertain which region of the CPC performs which 
operationa. Thus, the flow chart should provide the programmer with an entree 
into the program as given in the CPC listing. 

Although substantial opinion favors flow charts with gross detail, or no flow 
charts at all, carefully conceived requirements may demand detailed flow charts. 
If such is the case, alternative means of obtaining or satisfying the require- 
ment should be examined. The possibility of obtaining the flow charts by 
means of an automated flow chart system (i.e., a special purpose computer 
program) is certainly worth considering. Another possibility could be the 
substitution of a form of logic flow table instead of flow charts. The 
significant advantages of either of these approaches must lie in the ease and 
resultant economy with which they may be employed both in the initial pre- 
paration of the flow charts (or substitute) and their subsequent modification 
or updating. 

An automatic flow charting capability was considered in the BUIC III develop- 
ment. However, because of development costs and other factors including the 
detail of flow charts, the proposed capability was not deemed to have signi- 
ficant advantages over manual preparation and it was therefore rejected. This 
does not invalidate the concept as a potential approach for other computer 
program systems; it remains a possibility worth considering. In essence, each 
CPC is described three times within the Part II CPC volume: 1) prose; 2) flow 
chart; 3) listing. 

The reason for concern with regard to the level of detail presented in the 
CPC prose description and flow chart is not only the effort necessary to 
initially prepare the prose and flows, but the effort that must be expended 
to keep them current. A typical schedule for initial Part lis might be as 
follows: 

1) 60 days devoted to preparation of prose and flow charts 
by programmers 

2) 30 days for final typing, publication and delivery to SPG 
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3) 30 days for SPO examination preparatory to FACI. 

Program development consisting of coding and testing is normally underway con- 
current with the preparation of the Part lis. Moreover, both tasks are per- 
formed by the same programmers. Consequently, the level of detail presented 
in the prose and flows can be of considerable significance since programmers 
often do not firm the fine details of program design until coding and initial 
testing has been done. If the fine detail must be firmed 120 days prior to 
FACI, it may impose undue constraints on the program development process con- 
tinuing to FACI. It is recognized that this argument is not always true; 
difficulties can be minimized if sufficient allowance is made in the schedule 
for this task, but fine detail can be costly. Furthermore, experience has 
shown that com mand and control systems typically undergo considerable change 
after initial development. The changes may be necessitated to correct pro- 
gram errors, remedy design deficiencies, or to implement new requirements. 
Consequently, the greater the detail in the prose and flows, the more likely 
they will need to be changed to maintain identity with the computer program as 
the computer program undergoes change. Obviously, the cost must be evaluated 
against the benefits to be derived. 

Accordingly, the prose can describe in normal English such information as the 
operational tasks of the CPC, timing considerations, spares considerations, 
and unique or unusual aspects of the CPC design; the flow chart can identify 
the region of the CPC that performs each operational task; and lastly, the 
listing represents the actual configuration of the CPC accurately, identically, 
and completely. The prose and the flow chart then complement one another and 
provide a means of informing a programmer of the unique algorithms implemented 
by the computer program as well as where in the listing he should look for 
detailed information. Ultimately, it is the listing to which a programmer 
must look in order to find exact information of how the CPC performs its 
operations in terms of actual machine instructions. 

The chapter on interfaces in the BUIC III Part II CPC-oriented volumes, cor- 
responds to subparagraph 3.2.1.3 of the Form 9. These chapters did not, by 
themselves, satisfy the interface requirements. Since this was a subject 
pursued at some length in the FACI meeting conducted in May 1968, it is 
evident that mutual understanding by all responsible parties had not been 
previously attained. The question was essentially whether or not the inter- 
face requirements had to be treated in detail within each CPC volume such 
that each volume would be self-contained, or if the interface requirements 
could be satisfied by reference to other Part II volumes containing the de- 
sired information. The former approach would have included all interface 
information relevant to each CPC in the individual CPC volumes. This would 
have entailed extensive duplication of information in the Part lis as a whole 
which would not have particularly enhanced their value. Final resolution of 
this question was obtained by agreement that references would suffice. That 
is, the desired information was contained in Part II volumes such as Volume 4, 
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Compool and Layout, Volume 3, Compool Description, and Volume 7, System Item 
Set/Use Matrix; additionally, other CPC volumes describing subroutines are 
referenced as necessary. 

The approach taken in resolution of the question appears to be very satis- 
factory and reasonable. The required interface information is contained in 
the Part II specification in a form that can be readily understood by competent 
programmers. It had been suggested that inclusion of the interface detail in 
the CPC volumes would have benefitted inexperienced programmers charged with 
the responsibility of program maintenance. This argument is not necessarily 
valid; in the case of BUIC III even inexperienced programmers will need to 
know how to use Compool Description (Volume 3), set/use listings, and the like 
in order to fulfill their responsibilities. Thus, the interface requirements 
were satisfied by the special purpose volumes in conjunction with the CPC 
volumes . 

Of greatest significance with respect to the BUIC III Part II specifications 
is the question, "How much of v^hat was prepared is necessary and useful?". 
It was the consensus of contractor personnel that the net result of following 
Form 9 requirements to the letter was a level of detail below that which would 
be useful, and that subsequent specification maintenance at that level would 
prove to be impractical. 

This experience points up the need for good judgment in determing a proper 
level of detail to be required for the Part II specification. In general 
the two important objectives to be kept in mind are (a) its function as 
an instrument for configuration control and (b) its use by computer program- 
mers as an aid in diagnosing malfunctions and designing modifications. 
Since configuration control needs are met most directly by the actual listings, 
the prose and flow chart requirements should be directed towards attaining 
that level of description which will be adequate to convey an understanding 
of the design rationale, nature of design solutions, and the flow of information. 

The appropriate level of detail can be expected to vary for different computer 
programs and systems, as a function of such factors as computer program nature 
and complexity, expected frequency and magnitude of later design changes, and 
provisions for operational phase computer programming support and specifica- 
tion maintenance. In the usual case, it is believed that the minimum detail 
required to accomplish the purposes outlined above should also be viewed as a 
desirable maximum, since the expense of not only producing the specification 
initially, but of maintaining it later, increases significantly as additional 
detail is incorporated, particularly in the flow diagrams. 
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C. SPECIFICATION CHANGES 

It was pointed out in the preceding chapter that procedures for controlling 
changes to the specifications had been developed and applied during the earlier 
SAGE and BUIC II programs. As regards visibility by the SPO Configuration 
Control Board (CCB) , the earlier procedures had not involved formal processing 
of changes at the Part II level, but had been confined to the Part I (former 
"ops spec") level. However, experience with the earlier procedures had 
created a sufficiently firm base that the BUIC III configuration control and 
specification maintenance processes were handled efficiently, with relatively 
minor difficulties. 

Changes were initiated by issuance of a Preliminary Engineering Change Proposal 
(PECP) to the CPCEI specif ication(s) involved. In cases where the System 
Specification was affected, a PECP was accompanied by a Specification Change 
Notice (SCN) to the System Specification, but not by SCNs containing proposed 
word changes to the CPCEI specif ication(s) . Instead, the PECP described the 
change in sufficient detail to provide a basis for SPO review, approval, and 
contractual authorization. Following approval, the precise word and/or data 
changes were then developed and incorporated into specification change pages 
which were subsequently issued with covering SCN and formal ECP.* The volume 
of change pages was such that they were normally issued periodically in pack- 
ages, each package reflecting changes resulting from a number of ECPs and CRs 
(Change Reports, for Class II changes). 

Following the history of their predecessors in SAGE and BUIC II, the BUIC III 
computer programs were controlled primarily in terms of changes which were pro- 
cessed at the design requirements level, both before and after establishing 
product configuration baselines. Class I changes to the Part II specification 
only were limited to a few instances in which formal changes were processed to 
cover recompiling of the computer program instructions. 

BUIC III also followed well-established precedent in being characterized by 
frequent and extensive changes to the computer programs throughout Acquisition, 



The process described here represents a basic aspect of configuration con- 
trol for computer programs as embodied in ESD Exhibit EST-1, which departs 
significantly from the accepted change processing practice for items of 
equipment. Contractual authorization is necessary prior to the issuance of 
SCNs for the reason that the costs associated with computer program changes 
are totally matters of R&D, unaffected by such factors as production and 
logistic support. Costs of issuing exact changes to the Part I specifica- 
tion are normally appreciable; once the product baseline has been established, 
the total cost of implementing the change must have been incurred by the 
time all specification changes are completed. 
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Tables 6-1 and 6-2 below list the numbers of Class I and Class II changes, 
respectively, which occurred in the Part I specifications during the period 
March 1966 through December 1968. Changes affecting the Part II specifica- 
tions only, which occurred after product configuration baselines were est- 
ablished (effectively, April 1968), are summarized in Table 6-3. 

Change page packages to the Part I specifications were issued at average 
intervals of about 6 weeks between May 1966 and November 1968. Figures 
indicating the volume of changes pages for two of the CPCEIs, ADP and SEP, 
during that period are listed in Table 6-4. For comparison, the table also 
lists the initial and ending (November 1968) numbers of pages contained in 
the specifications. Data relating to how much of the material on the pages 
was changed, or on what per cent of the pages were actually affected, are 
not available. However, it may be noted that the new raw numbers are large 
enough that each of the original specification pages might have been replaced 
more than twice, during Acquisition. 

The reasons for changes were naturally many and varied. Also, the requirements 
for changes originated from a number of sources other than the contractor. 
While the information is not complete, sources are often identified in the 
"ECP Detailed Status Summary Forms" which were contained in periodic configura- 
tion management reports, giving descriptions and detailed status summaries for 
each ECP. These included NORAD, ADC, MITRE, Burroughs, and the SPO. In all, 
nearly 50% were positively identified as originating from such sources, largely 
based on needs arising from changes in interfacing equipment and systems. 



Table 6-1. ECPs to the Part I Specifications 



Computer 










Program 


1966 


1967 


1968 




ADP 


A6 


6 


17 




SEP 


26 


2 


1 




UCP 


15 


2 


1 




Totals 


87 


10 


19 = 


116 
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Table 6-2, Change Reports (CRs) for the Part I Specifications 



Computer 
Program 


1966 


1967 


1968 




ADP 


37 


42 


42 




SEP 


24 


63 


15 




UCP 


5 


9 


2 




Totals 


66 


114 


59 = 


239 



Table 6-3. Part II Specification Change - 1968 



Computer 






Program 


ECPs 


CRs i 


ADP 


1 


305 


SEP 


2 


167 


UCP 


1 


46 


Totals 


4 


518 



Table 6-4. ADP and SEP Part I Specifications: 
Number of Pages vs. Change Pages 



Computer 
Program 



1966 



1968 



Change Pages 



ADP 
SEP 



2192 
898 



2046 
866 



5762 
2682 
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D. DESIGN REVIEWS AND FACI 

1. Preliminary Design Review 

The Preliminary Design Reviews for the three BUIC III CPCEIs, ADP, SEP, and 
UCP, are most noteworthy in that the meetings did achieve their purpose; a 
formal technical review of the proposed basic design approach was satisfacto- 
rily accomplished. For each CPCEI, the documented preliminary design pre- 
sented information necessary to the conduct of the formal review. The infor- 
mation included a brief review of the interfaces identified in the Part I 
Specifications, a discussion of timing, a concise description of the CPCs, 
storage allocation, and other information pertinent to the computer program 
design. This was essentially in accordance with system management guidelines 
contained in EST-1 and, more recently, EST-3. 

The ADP PDR was significant in that it led to the subsequent cycle time analysis. 
The expected cycle time given in the preliminary design exceeded the minimum 
specified in the Part I Specification. Since the input/output table size was 
predicated on the specified minimum time, a determination was made that the 
I/O table design should be reconsidered to make more efficient utilization of 
the output capabilities. The ensuing analysis revealed the cycle time problem. 

Also of interest is the fact that, contrary to the understanding obtained at 
the ADP PDR, JOVIAL was not used as projected for the development of ADP. It 
was intended that new CPCs or rewritten BUIC II CPCs be written in the JOVIAL 
computer program language. Subsequent efforts revealed that the expansion 
factor prohibited the use of JOVIAL; code written in JOVIAL could not remain 
within the storage constraints. Consequently, a decision was made to forego 
JOVIAL for ADP and use machine language instead. 

Thus, it is evident that the PDR served as the stage for involving the SPO/ 
MITRE monitors in the decision-making process with respect to significant 
aspects of CPCEI program design. The design agreed to at the PDRs facilitated 
mutual understanding by SDC and SPO/MITRE of the problems and their solutions 
in the computer program development that followed the PDR. 

2. Critical Design Review (CDR) 

As described at an earlier point (Chapter V) , CDRs were scheduled and held 
incrementally, and for the most part in conjunction with preliminary quali- 
fication testing. While there were some discrepancies associated with the 
functional emphasis of PQTs as opposed to the computer program component (CPC) 
emphasis of CDRs, the reviews were generally concerned with design documen- 
tation for the elements undergoing test. Hence, the reviews were held at the 
level of completed design for those elements. 
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Despite their close association with PQTs, however, and in contrast with the 
PQT experience (see Section F below), there is a notable absence of reports 
of difficulty, resulting changes, or other action items related to the conduct 
of CDRs. In general, the CDRs occasioned very little concern or comment. 

On the whole, these BUIC III CDRs did not appear to involve elements which 
were particularly novel as compared with prior practice. They were often 
attended by SPO representatives, and typically included presentations of the 
design, functions, timing, size, and interface characteristics of the CPC(s) 
being reviewed. However, beyond the point of providing visibility of technical 
progress, the purposes apparently did not extend to reaching significant 
decisions with regard to the development cycle of the computer programs involved, 
This is perhaps not an unexpected circumstance, considering that the designs 
were already complete at the time of reviews. 

In Exhibits EST-1 and EST-3, the emphasis is placed on CDRs which are held 
''at the level of flow charts or computer program logical design prior to 
coding and testing." At that level, the stated formal objective is to 
identify the design documentation which will be released for coding and 
testing. Experience on computer program CDRs in that category is not avail- 
able from BUIC III or other system programs with which the authors have been 
associated. However, computer programmers have volunteered firm opinions to 
the effect that it would not have been feasible to accomplish the stated ob- 
jectives of a CDR at that level for a CPCEI as complex as the ADP. Comments 
are made that: the design which initiates coding must remain flexible during 
the coding process; exact interfaces among CPCs are not visible for review on 
an incremental basis; and the ability to conduct such reviews in a meaningful 
and adequate fashion would require significantly greater technical resources 
than the SPOs have typically had at their disposal. 

While the interim design level (i.e., at flow charts) is given the primary 
emphasis, it is to be noted that EST-1 does provide a range of options which 
is broad enough to cover the BUIC III application.* Judging from the comments 
which have been made, and the fact that they did not contribute additional 
problems, it may be fortunate that the CDRs were handled as they were, in 
BUIC III. 

3. First Article Configuration Inspection (FACI) 

The first Article Configuration Inspection of the three CPCEIs was essentially 
accomplished in May 1968. The FACI was noteworthy in that there was very 
little precedent for conducting such an inspection for computer program items 
in accordance with recognized system management concepts.** While the FACI 



* ESD Exhibit EST-1, Section H, p. 40-13. 
** As noted elsewhere, one element of SEP had been FACI'd in September of the 
preceding year. 
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did not proceed without some problems, the nature of the problems did not pose 
severe obstacles and they were resolved to the mutual satisfaction of the part- 
icipants. The objectives of the FACT were achieved; the Part II specifications 
were audited and approved with the qualification that revisions based on SPO 
comments be incorporated in the basic issues of the Part lis which were due to 
be published by 1 February 1969. With SPO approval, the Part lis were recognized 
as the instruments defining the product configuration baseline for the re- 
spective CPCEIs. For practical purposes, however, the baseline had been 
established in the preceding month, April, at the onset of Formal Qualification 
Tests. Formal control was maintained of all changes to the draft Part II speci- 
fications which had been issued describing those FQT configurations. 

FACI was preceded by FQT and followed by Category II testing. This relative 
timing is reasonable and conforms to established concepts. However, certain 
deviations did occur as a result of the known cycle time deficiency in ADP. 
Although an approach to correct the deficiency had been agreed upon and approved, 
its implementation had not been completed prior to FQT and FACI. Since the 
corrections entailed extensive changes and recompilation of five computer 
program components, SDC and SPO/MITRE agreed to defer the FACI audit of the Part 
II volumes for those five CPCs. The audit of those five CPC volumes was then 
performed in February 1969 after implementation of the cycle time changes, thus 
concluding the FACI. An unusual aspect of those five volumes is that the basic 
issues include Specification Change Notices for ECPs and CRs that were pro- 
cessed against previous draft Part II volumes of the five subject CPCs. 

The contrast of the approach taken to the FACI meetings for BUIC III and SEEK 
DAWN Interface Computer Program (SDICP) is interesting. In both cases, draft 
copies of the Part II specifications were delivered to the SPO in advance of 
the FACI meetings; BUIC III drafts. were delivered approximately 30 days before 
FACI; SDICP drafts were delivered approximately 90 days before FACI. Whereas 
the detailed review of the BUIC III Part lis was performed at the FACI meeting, 
the review of SDICP Part lis was essentially done prior to the meeting. In 
the latter case, SPO/MITRE comments were sent to SDC so that mutual agreement 
had been substantially obtained before the FACI meeting. The meeting thus 
served to confirm understandings that had already been reached and to resolve 
those few questions that had not been completely resolved earlier. 

The difference in approach is of interest since the one week FACI meeting for 
BUIC III (disregarding the second meeting for the five deferred CPCs) permitted 
only a superficial examination of the Part II specifications. It is highly 
likely that a more comprehensive examination of the BUIC III Part lis would 
have revealed many more discrepancies among the CPC descriptions, flow charts, 
and listings. Depending on the bulk of the Part II specifications, available 
SPO technical resources, and other relevant factors, the SDICP approach may be 
the more desirable one in conducting FACI on other CPCEIs. 

The prospect of a FACI, and resulting formal control at the level of product 
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configuration, was viewed in advance with some apprehension. While procedures 
for computer program control and accounting had been in effect for systems pre- 
ceding BUIC III, they were developed and used internally by the contractor, had 
been largely confined to control of computer program listings, and had not been 
carried out under the formal label of configuration management. However, with 
minor exceptions, anticipated difficulties did not materialize. 

There were two areas of potential difficulty which deserve mention. The first 
related to detailed flow charts for CPCs, which proved to be expensive and 
time-consuming to maintain in the draft Part II specifications during a period of 
some months preceding FACI. The possible continuing impact of this problem 
was avoided, following FACI, by a SPO-approved change which eliminated the CPC- 
level flow charts from the specification. 

A temporary problem arose in the course of Category II testing, relating to a 
proposed requirement for a new Version Description Document to cover each 
daily change made in connection with the test activities. The question of what 
constitutes a "new version" is not unequivocally defined, and disagreements 
occurred on the point among personnel of the SPO, contractor, and test team. 
It developed that the real issue was more a matter of test philosophy than of 
control procedures, and working solutions were reached following a week or two 
of discussion. 

However, the BUIC III experience as a whole has provided clear evidence that 
the configuration control and accounting procedures which had been adapted for 
computer programs were remarkably efficient. Some data relating to rates of 
changes were presented in the preceding section of this chapter. Following 
FACI, control was extended to cover the computer programs and Part II specifi- 
cations, and was maintained routinely under complex circumstances. In practice, 
situations arose in which as many as three versions of the Air Defense Program 
were in existence or under development at one time, each differing signifi- 
cantly from the others with respect to incorporation of approved changes and 
scheduled introduction into test or operational use. The process included, 
for example, accounting for Class II error corrections made to a given version 
which were either applicable to succeeding versions or not applicable because 
of superseding Class I changes, and also involved keeping track of change 
relations with other computer programs, support documentation as well as 
specifications, and items of system equipment. Under these conditions, con- 
trol was maintained at all times and with relatively minor difficulties. 
Considering the frequency and volume of changes implemented, it seems clear 
that the BUIC III experience has also demonstrated that post-FACI control need 
not seriously impair the flexibility with which computer programs can be 
altered to accomplish desired changes in system functions. 



86 



E. CATEGORY I TEST PROGRAM 

When the Category I test requirements were introduced at the outset of the 
BUIC III program, they were generally viewed as innovations which would have 
the primary effect of merely formalizing various aspects of the **normal** test 
process. Computer program test and evaluation (CPT&E) , although introduced for 
the first time as a new term, was readily recognized as an inherent and 
familiar part of computer program development. It was not immediately obvious 
in advance that the addition of a few preliminary and formal qualification 
test sessions would significantly affect the total computer program acquisition 
job to be accomplished. Nevertheless, Category I testing proved to be an 
expensive and troublesome area throughout the BUIC III effort, in essentially 
all of its various aspects. 

The record indicates, as reviewed in the preceding chapter, that the Category 
I test milestones — in the form of PQTs and FQTs and their various associated 
documents — were actually accomplished during the program. This achievement 
now seems a remarkable one, at face value, although it is conceded to have been 
made possible only by the fact that the system schedule slipped appreciably, 
for independent reasons, during the critical periods. Throughout, difficulties 
and complexities were faced by SPO and contractor personnel in interpreting, 
coordinating, and implementing the many requirements. In fact, the effect of 
resulting confusions and misinterpretations among personnel involved in the 
program was so pervasive that an unbiased and factual account of the problem 
is difficult to reconstruct on the basis of available records and verbal 
reports. 

Preliminary qualification testing of the ADP is the specific area in which 
troubles are universally reported as being most important and acute. As out- 
lined earlier, a total of 35 PQT sessions were conducted on 19 different dates 
during the 15-month period of September 1966 to February 1968. Associated with 
these 35 PQTs were 53 PQT Procedures documents and 63 Test Reports. Although 
the reasons for these discrepancies are not all accounted for, it appears that 
some PQTs had multiple objectives for which separate procedures were written, 
some PQTs failed and had to be re-run, some of the reports had to be re-written, 
and a few scheduled PQTs were never accomplished. 

The objective established for PQTs in the Category I Test Plan was to verify 
approximately 2500 **reactions** of the computer program in conformance with 
detailed requirements which had been set forth in the Part I specification. 
PQTs were planned, as a complete set, to provide comprehensive coverage of 
nearly the whole of the Section 3 requirements for the ADP CEI. As later 
described by contractor participants, some (i.e., most) of the PQT sessions 
were planned, rehearsed in advance, and then conducted as "demonstrations," in 
effect, for the benefit of visiting SPO personnel. These were generally success- 
full, but were also time-consuming and more costly in manpower than had been 
anticipated. Others were prepared but not rehearsed, and were actually run 
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in conformance with the written procedures for the first time during the for- 
mal PQT meeting. These were more in the nature of true "tests" but they 
often failed and had to be re-run at subsequent times. The program as a whole 
is reported to have caused cost overruns and difficulties in meeting schedules, 
as well as significant morale problems among many computer programmers, who 
came to regard the PQT requirements as monstrous and artificial barriers to 
their progress towards completing the ADP development task. 

The 35 PQTs which were carried out did manage, in one way or another, to result 
nevertheless in formal verification of all but 450 of the original 2500 re- 
actions prior to the beginning of FQT and Category II testing. Of those, all 
but 110 were subsequently verified at the Category II site, and the remaining 
few were eventually waived. 

Essentially the same problems were encountered in PQTs for the system exercising 
CPCEI (SEP), although the totals of events and documents were slightly less 
numerous: 24 PQTs, 30 Procedures documents, and 27 Test Reports. In the 
utility area (UCP) , qualification testing for the entire CPCEI was confined to 
one "PQT" for each of two (out of a total of 25) of the elements: except for 
these two tests, the entire CPCEI was qualified in relatively painless fashion 
through its use in supporting the Acquisition phase development of ADP and SEP. 

The formal qualification tests (FQTs) for ADP and SEP were held, as noted 
earlier, in a number of separate test sessions in each case. For SEP, the 
FQT was "incremental," in the sense that the total consisted of separate test 
sessions for each of the major elements of the CPCEI, each having a separate 
associated plan, procedures, and report. This arrangement was made possible by 
the fact that those major elements of the CPCEI represent capabilities which 
can operate independently. In the case of ADP, four separate FQT sessions 
were run to examine separate aspects of the performance of the integrated 
CPCEI . * 

The Procedures documents for FQTs posed additional difficulties which had not 
been typical of PQTs. Whereas PQT Procedures had been delivered only for SPG 
review, normally about 30 days in advance of a scheduled test, the FQT Pro- 
cedures were required a minimum of 90 days in advance for review, approval, 
and revision based on SPG comments. The records indicate that initial drafts 
of some of the FQT Procedures for ADP were revised and reissued as often as 
five times before being accepted, because of various problems involving con- 
formance to the Form 9, the style of presentation, and technical adequacy. 
This, as in the conduct of PQTs, was an area in which the project personnel 
encountered an unexpectedly firm insistence by the SPG on rigorous adherence 
to the requirements, and in which inevitable difficulties resulted from a 



* For both ADP and SEP, all test sessions were run on an identical configura- 
tion of the CPCEI. 
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general absence of adequate guidance and experience. However, the necessary 
experience was acquired in the course of this troublesome process, to the 
degree that a subsequent FQT (see below) was documented and accomplished with 
significantly fewer perturbations. 

The FQT for the first version of the ADP was accomplished immediately prior to 
Category II testing, at the Category II site. This timing and location had 
been anticipated from the outset of the program, based on a variety of con- 
siderations which have been recognized as being generally pertinent to the 
qualification of such complex operational computer programs as the ADP.* How- 
ever, as was noted in the preceding chapter, a second FQT was performed (the 
Timing Version) approximately four months following the completion of Category 
II testing, and a third and fourth are planned for additional follow-on 
versions which are under development and scheduled for delivery at future 
dates. The initial version of ADP which underwent FQT prior to Category II 
testing contained a cycle-time deficiency. Interim corrections were devised 
for Category II use, and the deficiency was removed fully in the Timing Version, 
for which FQT was then repeated. Thus, one might assiime that qualification of 
the CPCEI had been accomplished by that point in time. Yet, future FQTs are 
being planned for the follow-on versions.** 

While the continued repetition of FQTs is apparently not regarded with any 
particular concern by either the SPO or contractor, it does suggest that there 
are certain questions regarding the philosophy of qualification as applied to 
computer program contract end items which remain to be clarified. Judging 
from SAGE/BUIC II precedents, the ADP can be expected to undergo continuing 
changes which will probably occasion periodic new versions at the rate of 
three or four per year, each containing additional, altered, and/or deleted 
capabilities as compared with the version being replaced. In SAGE, each new 
version is subjected to formal tests which are comparable in scope and level 
to the "FQTs" planned for follow-on Acquisition phase versions of the BUIC III 
ADP. These SAGE tests, however, which are actually conducted by the using 
command prior to introducing each new version into operational use, are known 
as "acceptance" tests. One might question the application of either term, 
acceptance or qualification, to such tests — considering that "qualification" 
is normally confined to an initial article, while "acceptance testing" is 
firmly associated with individual units of a CEI undergoing quantity production. 



* The considerations are mentioned and discussed more fully in ESD Exhibit 
EST-1, ESD Supplement 1 to AFSCM 375-4, and ESD-TR-68-1. 

** It is being suggested here only that this situation poses some novel ques- 
tions relating to the general concepts of qualification and acceptance 
testing, not that the repetition of FQTs was in any sense "wrong"; in fact, 
it was clearly an effective and appropriate procedure for the circumstances. 
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This appears to be an area in which the type of computer program represented 
by ADP may call for some new testing concepts which are not readily derived 
from established practice with equipment items. 

It was mentioned above that the objective of PQTs was to verify some 2500 re- 
actions of the computer program, covering the spectrum of detailed requirements 
set forth in Section 3 of the Part I specification. While it is to be 
assumed that the computer program must be designed to meet those detailed 
requirements, and must "work" in the operational system, the sheer number of 
reactions alone suggests that the task of formal verification can be one of 
considerable magnitude. Many of the reactions are interdependent; they should 
occur under variable combinations of input and operating conditions, and many 
can be verified only through intricate analysis. This much was evident in 
varying degrees to both SPG and contractor people who were involved in planning 
the Category I testing at the outset of the BUIC III program. It was also 
obvious at that time that the total job could not be accomplished during a 
limited period of formal qualification testing at the Category II site. Based 
on the ground rule that the comprehensive formal verification had to be done, 
however, it was decided to place the burden of verifying detailed reactions 
onto the PQTs, and to limit FQT objectives to the testing of higher-level 
characteristics of the integrated CPCEI operating under system conditions. 
This decision was reached early enough to influence the writing of the Category 
I Test Plans for ADP and SEP. Together with some lack of understanding of the 
test plan functions in relation to Section 4 of the Part I specification on 
the one hand, and to the subsequent Procedures documents on the other, it 
accounts for much of the bulk and detail in the Category I Test Plans. 

Thus, the groundwork was laid for the troublesome experience which ensued in 
trying to carry out the total of 57 PQTs for ADP and SEP. It became recognized 
early in the process by both SPG and contractor, although for different reasons, 
that the attempt to accomplish formal qualification by means of PQTs was 
unsound — to the contractor because the magnitude of the task has not been 
appreciated and reflected in planning manpower and schedules, and to the SPG 
because of an emerging realization that such testing accomplished during 
intermediate stages of evolving those complex and massive CPCEIs would provide, 
at best, a weak basis for guaranteeing the presence of specified reactions in 
the "final" product. However, the contractual commitments reflected in the 
test plans were enforced, except for the few reactions which were eventually 
waived after Category II testing and two FQTs for each of the major CPCEIs had 
been completed. 

The test process was also obviously affected by the continuing flow of Part I 
specification changes throughout the period during which PQTs were being con- 
ducted, as well as by coding changes in PQT'd elements as they were assembled 
and made to work with other elements of the computer program. The desirability 
of baselining PQT'd elements at the instruction level was considered briefly 
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at an early point, in the context of qualifying via PQTs, but was discarded as 
being an impractical requirement to implement. The record does not show how 
many PQTs, or tests of Individual reactions, were vitiated by subsequent Part 
I specification changes, or how many were added as new requirements, although 
it may be inferred from the number of Category I Test Plan changes listed in 
the Configuration Index — as well as from the voluminous flow of Part I 
specification change pages resulting from ECPs — that the numbers were sub- 
stantial. These changes were known and clearly recognized during the program. 
Also, in some manner which was never readily visible to an outside observer, 
the contractor was able to implement the changes and maintain an updated test 
effort which was at least adequate at a practical level; internally, similar 
change rates in SAGE and BUIC II had become a way of life. However, there is 
evidence now that an overt recognition of this "fluid" nature of the computer 
programs in question was conspicuously missing at the time the PQTs — and the 
FQTs too, for that matter — were initially being planned in BUIC III. In re- 
trospect, it appears that the testing approach adopted would have been more 
fitting for CEIs having less complex performance requirements, which had more 
stable definitions at the outset in terms of Acquisition phase end objectives, 
and which could be expected to have relatively longer operational lives in 
their FACI'd configurations. 

One may assume, as most of the BUIC III personnel actually did, that the trials 
and tribulations of Category I testing were caused by application of the new 
375-series requirements. This was undoubtedly true in a sense, although a 
step-by-step study of the requirements — i.e., of the Forms 9 and exhibits — 
fails to reveal any significant errors or unsound provisions which can be 
pointed to as being responsible for the troubles. In fact, it seems clear that 
the major difficulties were a direct result, not of their application, but of 
their misapplication. PQTs, for example, should never have been planned in 
the first place as formal tests which would actually qualify the CPCEIs with 
respect to the full ranges of requirements contained in their Part I specifi- 
cations. 

However, the BUIC III experience makes it equally clear that the existing re- 
quirements are inadequate, in and of themselves at least, to guide a success- 
ful Category I testing program for such CPCEIs as the ADP and SEP, As 
mentioned earlier, requirements were formulated and communicated in the legal 
language of Forms 9 and contractual exhibits, with a minimum of explanatory 
material or other guidance to their specific interpretation as applied to the 
CPCEIs in question. By and large, the requirements are sufficiently general 
in wording to cover many other types of computer programs, including those 
which would be far less complex and would remain relatively stable during both 
Acquisition and Operational phases of their existence. By virtue of this very 
generality, however, they are deficient in both guidance and specific coverage 
for the classes of computer programs represented by ADP and SEP. Although it 
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is beyond the purpose of this report to construct a detailed analysis and pro- 
pose specific changes, the points discussed briefly below will attempt to 
identify the problem areas more directly, and to suggest better interpretations 
which might have been made to the requirements as they now stand. 

1. The degree to which formal verification of performance requirements 
must be accomplished during Category I testing is a matter which poses special 
problems in the case of many CPCEIs. Existing guidance tends to imply that all 
requirements of Section 3 in the Part I specification must be satisfied. Yet 
it can be shown that for the ADP, for example, if one takes into account the 
numbers of defined characteristics and their countless potential interactions 
during operating conditions, complete verification would be virtually impossible 
to achieve, even if no changes were permitted for the life of the item. Hence, 
a contractor cannot reasonably commit himself to the task of proving complete 
verification. At the same time, a SPO obviously needs assurance that the com- 
puter program does meet its requirements, at least well enought to permit 
successful operation of the system. This dilemma was not clearly recognized 

in BUIC III, and many of the difficulties can be traced to the supposition 
that complete verification could actually be accomplished. Pending the 
development of better guidelines in this area, the problem seems to indicate a 
need for good judgment in the SPO-contractor relationship to assure that 
qualification requirements are established on a practical and cost-effective, 
rather than perfectionist, basis, taking account of both the CPCEl's inherent 
complexity and its anticipated degree of stability as an element of the 
operational system. 

2. The meaning and purposes of preliminary qualification testing were 
interpreted in varied ways by personnel associated with BUIC III. Some of the 
confusion might be ascribed to the term itself, which implies qualification as 
a purpose; and the available guidance is sufficiently scant that it has to be 
analyzed carefully to derive the intent that PQTs are really preliminary _to^ 
qualification. Beyond this bare interpretation, official documents presently 
provide only meager amplification — although there are some isolated re- 
ferences which might be used as starting points. For example, the following 
statement is contained in ESD Exhibit EST-1, Section H:* 

"Since the total process [i.e., of computer program development] 
is typically lengthy ..., and represents the major expense of 
computer program acquisition for the system, it should normally 
include preliminary qualification tests/demonstrations at 
appropriate stages for formal review by the procuring agency. 
Requirements for such tests are established in Section 4 of the 



* Section H is a portion of EST-1 which did not appear in the BUIC III con- 
figuration management exhibit. 
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Part I specification and are amplified in the contractor's 
Category I Test Plan. While the tests are preliminary in 
nature (they do not imply acceptance, or formal qualifi- 
cation), they can serve the necessary purposes of providing 
check points for monitoring the contractor's progress towards 
meeting design objectives and of verifying detailed per- 
formance characteristics which, because of sheer numbers 
and complexity, may not be feasible to prove in their 
entirety during a limited period of formal qualification 
testing." 

Elsewhere, an ESD technical report* presents the concept that PQTs can be re- 
lated to successive stages of assembly of CPCEI components during the develop- 
ment process. The authors illustrate a "building block" approach to CPCEI 
development, in which each new CPC is envisaged as being assembled to the 
existing piece (s) of the CPCEI and each assembly stage is preceded by a CDR 
(prior to coding the CPC) and accompanied by a PQT. In the authors' words: 
"As each computer program component is added and each PQT conducted, increased 
confidence develops in the CPCEI being tested." 

Such sources provide a basis for regarding PQTs as having primarily a "con- 
fidence-building," rather than qualification, purpose. It is suggested that 
the concept might be further amplified along the following lines: 

a. PQTs should be planned as formal events which will 
enable the SPO to verify contractor progress in 
developing a computer program CEI, prior to its 
formal qualification. 

b. A PQT should be regarded as a demonstration, rather 
than a true "test," since it is presumed that the 
contractor will have accomplished the necessary 
assembly testing for the components in question as 
part of his CPT&E effort. However, sessions could 
be arranged to provide SPO observers the opportunity 
to verify additional detailed requirements where 
indicated, perhaps on a sampling basis. 

c. Scheduling should be based on the contractdr's 
CPCEI development plan, such that PQTs are timed 
to coincide with stages at which key components, 

or assemblies of components, are due to be completed 



* Piligian, M. S. and Pokorney, J. L. , Air Force Concepts for the Technical 
Control and Design Verification of Computer Programs. Electronic Systems 
Division Technical Report, ESD-TR-67-67, April 1967. 
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at an operable level. Scheduling should be spaced, 
to provide visibility of progress at early, inter- 
mediate, and later points during Acquisition. 

d. There should be no necessary requirement to subject 
all components of a complex CPCEI to PQTs. Emphasis 
should be placed on components and assemblies which 
are critical and/or which will provide adequate in- 
dices of contractor progress in meeting the CPCEI 
schedule and design objectives. Considering time 
and cost, it is suggested that 3 or 4 judiciously 
spaced PQTs would have accomplished the purpose for 
the BUIC III ADP, instead of the 35 which were actually 
held. 

3. As noted above, the emphasis on PQTs in BUIC III resulted from an 
initial recognition that the FQT, which was planned (for ADP) to be held during 
a short period between the installation/checkout of computer programs at the 
Category II site and the beginning of Category II tests, would not be adequate 
to accomplish the full qualification objectives. The question of how those 
objectives can be met, other than by shifting the burden to PQTs, does not have 
answers which are altogether obvious. Consideration has to be given to the wide 
variations among CPCEIs which can exist with respect to such characteristics 
as size and complexity, stability of configuration once developed and criticality 
of performance characteristics during operational employment. 

Generalized requirements should therefore permit appropriate solutions to be 
formulated on a case-by-case basis. However, for computer programs like ADP 
and SEP, it is believed that, better solutions could have been reached within 
the framework of provisions which now exist, e.g., in ESD Exhibit EST-1 and 
elsewhere. It should be recognized that the initial burden of arriving at 
feasible and adequate solutions for a given CPCEI falls on the contractor, who 
must propose the solutions in the process of formulating test requirements to 
be contained in Section 4 of the Part I specification. Alternatives which 
might have been considered, but were not tried in BUIC III, include the 
following: 

a. The contractor is responsible for comprehensive testing 
against each and every performance requirement of the 
computer program, as part of CPT&E. If the development/ 
testing process is properly managed, it should be possible 
to utilize CPT&E as a source of data for qualifying much 
of the detail, in particular, which can be time-consuming 
and expensive to repeat during formal test/demonstrations. 
To exploit this possibility, the contractor should be 
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required to demonstrate the effectiveness of internal 
management controls, as well as to furnish data and/or 
other appropriate evidence that individual requirements 
are satisfied. Direct verification by the SPO of de- 
tailed requirements in selected areas could be carried 
out on a limited sampling basis during PQTs and FQT. 

b. An FQT which is held at the Category II site is necessarily 
limited. Appropriately, as in the case of ,ADP, it should 
emphasize testing of major requirements for the integrated 
CPCEI performing in operationally configured equipment with 
live inputs. However, there is reason to believe that it 
should have been feasible to accomplish some of the ob- 
jectives for ADP earlier. For example, much of the inter- 
face testing with the maintenance-diagnostic computer pro- 
gram, and the simulation mode test, would have been possible 
to accomplish in-plant, with proper advance planning. Such 
objectives as Live Mode and System Load definitely required 
the Category II facilities. However, to the degree that 
valid tests are made possible by available simulation, equip- 
ment, and other relevant factors, they should be accomp- 
lished by FQTs held in-plant. 



F. BUIC III CYCLE TIME PROBLEM 

1. Statement of the Problem 

The BUIC III "cycle time problem" is most simply described as excessive oper- 
ating time of the BUIC III Air Defense Computer Program (ADP). During ADP 
program development, analysis and testing revealed two significant aspects of 
the problem: (1) radar inputs processing; and (2) operating time of the com- 
puter program components that perform control, guidance, and display makeup. 

The System Specification, SS-ES 416M 65-lB, describes timing by alluding to 
it in the performance allocation requirements for radar inputs. In this con- 
text, a correlation period is specified in seconds; two correlation periods 
equal one radar scan. Operating time was specified in the Part I Specification, 
CGTM2385A, Volume 1, as seconds minimum to seconds maximum per cycle, and 
seconds minimum to seconds maximum per bicycle. The minimum value was an 
estimate based on the subframe time of SAGE; the maximum was an arbitrary 
value. It was assumed that if the maximum value were exceeded, an equipment or 
program malfunction had occurred such that the BUIC Confidence Diagnostic 
Program (BCDP) should operate. These values had been previously established 
for BUIC II and were then applied to BUIC III. 
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Estimated cycle time of the BUIC III operational program was initially derived 
by adding increments for the differences between BUIC II and BUIC III features 
to the cycle time of BUIC II which had been determined in a timely study em- 
ploying reasonably high load conditions. The differences were identified 
principally in input/output and computer program size. 

This information was documented in TM-2385/000/00, Preliminary Design of BUIC 
III Air Defense Program, and submitted at the Preliminary Design Review (PDR) 
held January 1966. At the time of the PDR, no difficulty was anticipated in 
meeting the cycle time constraints; however, input/output table size had been 
predicated on minimum cycle time. This table design made inefficient utiliza- 
tion of output lines with the possibility of system degradation due to loss or 
delay of information to be transmitted to interceptors and other facilities. 
It was determined that input/output table size should be based on projected 
cycle time. Accordingly, SDC was committed to a study of cycle time pro- 
jections considering I/O time, compression of input data to one word format, 
and the effect on I/O table structure. 

In the .months that followed the PDR, SDC did perform a study which led to a 
new prediction of cycle time. The study was performed using a load based on 
a moderate threat situation which entailed a given number of radar returns/ 
cycle, interceptor capacity, and 20 less than full track capacity. The re- 
sultant prediction as of the end of May 1966 was substantially greater than 
the earlier estimate submitted at the PDR. At this point, it was clear that 
the possibility of adverse effects on system performances was sufficient to 
justify continued scrutiny and evaluation. 

Radar Inputs Processing 

The continuing study ascertained that the operation of the radar inputs pro- 
cessing program (RIP) consumed a relatively large amount of time. RIP's 
operating time is a function of the number of radar returns processed, number 
of returns correlated, and other factors. As a necessary part of this study, 
the possible effects of high cycle time were determined and are briefly 
summarized here: 

a. The operation of ADP would be interrupted in each cycle 
exceeding a specified threshold number of seconds since 
ADP would assume a malfunction and initiate BCDP to 
investigate. 

b. Degradation of active and passive tracking functions could 
occur since tracking parameters had been optimized based 
on specified minimum cycle time. 
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c. Height reply rate could be less than required to delay 
in sending out height requirements, 

d. Response to operator switch actions could be delayed. 

e. Duration of forced displays would vary, thus possibly 
causing operator confusion. 

f. Guidance calculations for interception could be in error. 

Concurrently with the SDC study, MITRE engaged in a parallel study which also 
identified radar return processing as taking a large amount of operating time. 
Since both SDC and MITRE agreed, further study and analysis concentrated upon 
radar inputs processing, A solution was recommended to the SPO and approved 
that entailed a redesign of the radar inputs computer program components such 
that the average processing time for each radar return was reduced; additionally, 
a new program system design was developed that provided for operating, some 
functions less frequently than every cycle and also presented a new data 
limiting scheme. This change was implemented as ECP 54-1 and was included in 
the computer program when Program System Test (PST) began. 

Control 

PST of ADP was performed at the BUIC Evaluation Facility, Hanscom Field, 
Massachusetts, using live inputs to the computer program. The testing 
revealed that the cycle time problem had not been fully resolved; the earlier 
prediction of operating time for non-radar processing programs was not valid. 
It was determined that the underestimation was principally in the amount of 
time the control program (COP) and the CPCs performing guidance and display 
makeup took to operate,* 

BUIC II hardware had an output interrupt every 70 ms . that allowed the control 
program to transfer output messages as well as read the message processor. 
BUIC III hardware has an additional interrupt every 30 ms. for processing in- 
puts from the message processor. The high operating time of COP was a result 
of the high frequency of interrupts being processed by inefficient interrupt 
processing logic. The logic of COP included use of an executive loop which, 
when entered, would check indicators and items to determine the need to 
service any terminal device, the need to start any I/O operations, the need 
for action due to completion of an I/O operation or the need to process a 



* Since this problem is treated fully in TM-4153/001/00, Final Report of BUIC 
III Timing Analysis, 12/15/68, no attempt is made here to address the problem 
in its entirety. The discussion does address COP because of its unique 
aspects and significance. 
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clock interrupt or an external request interrupt. When any interrupt except 
an equipment failure interrupt occurred, the indication of the interrupt was 
set and the executive loop was entered. If, as in the case of message pro- 
cessor service, multiple l/O was required, several passes through the loop 
would be performed. Because of the large amount of code operated in this 
loop and the frequency of messages processor interrupts (one every 21 ms. in 
COP), the control program required about 15/21 of ADP operating time for 
message processor servicing. 

This problem was resolved by an extensive redesign of COP to make it more 
efficient and by changes to the message processor servicing interval (ECP 108) 
to reduce the frequency of interrupts. To reduce the amount of time required 
for interrupt processing the following changes were made in the control program 
design: Interrupts as much as possible are no longer queued, but are com- 
pletely processed when they occur. This reduces the amount of code operated 
when an interrupt is processed. Secondly, the number of interrupts per unit 
time is reduced. The program now services a clock interrupt at a fixed fre- 
quency of once every 30 milliseconds with message processor servicing per- 
formed on every other clock interrupt. The routines with high frequency of 
operation were designed and coded with time efficiency the main concern. This 
ECP was included in the Timing Version subjected to Formal Qualification 
Testing at the BEF 2 December - 13 December. The FQT indicated that the 
Timing Version conforms to specification. 

2. Conclusion 

The cycle time problem insofar as radar inputs processing was concerned was 
identified early in the development of ADP and resolved within a reasonable 
time; of greater concern is the late realization that a cycle time problem 
still existed that had its origin in the control program (COP) . That the 
latter should remain undetected until PST must be regarded as a deficiency 
in the activities preceding PST. Testing during CPT&E of COP did not 
sufficiently approximate its exercise in a live environment, the difference 
being principally in the lack of live inputs that would actuate the processing 
routines of COP. Hindsight indicates that CPT&E should include a full 
exercise with attention to the timing of the control program, as well as the 
other CPCs comprising the CPCEI, in a manner equivalent to employment live. 
This means that special purpose program tools must be designed and developed 
to facilitate such testing. Timing tools that were used in the control pro- 
gram study, redesign, and testing are described in Table 6-5, 

Experience has shown that two critical factors must be considered in program 
design: timing and storage. Furthermore, each must be considered in relation 
to the other. Although operate time is basically a direct function of program 
size, an inverse relation may also exist. For example, if operate time had 
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Table 6-5. Timing Tools Description 

Ratio of COP to NON COP Time . (COP + non COP = total ADP) . 
This tool was developed to monitor the Item N005 In COP. 
This Item Is greater than whenever COP Is operating. The 
software routine that was utilized operated In the non-ADP 
computer and generated counts of when ADP was In COP and 
out of COP. 

Ratio of Message Processor Time Consumption to Total ADP . 
Real time Clock Interrupts were blocked and a program loop 
was placed In the Manual Inputs Program (MIN) . The Item 
N005 In COP was again monitored for a value greater than 
7 to determine the percentage of time required for message 
processor servicing. The software routine that was developed 
operated In the non-ADP computer. 

Interrupt Count and Frequency Recorder . Two tools were 
developed to determine the frequency and Interrupt mix In 
ADP. Both tools utilized were developed by octally modifying 
the control program to generate counts per cycle or to save 
the Index register setting (XIR) . The XIR was saved for an 
Interrupt mix and sequence analysis. 

Increases In Cyclic ADP Due to Interrupts . A timed loop was 
placed In the Telling Program (BIT) - As additional Interrupts 
were allowed to occur the operating time of BIT was recorded 
by a TIMER program In the non-ADP computer. The difference 
between the expected operating time of BIT (that of the timed 
loop) and that actually recorded Indicated the time Increases 
due to Interrupt processing. 

(Word counts of) Message Processor Input Operations . A routine 
was generated that recorded the result descriptor of message 
processor Inputs. The differences between the maximum message 
processor word count and that of the actual result descriptor 
defined how many words were actually transferred. 
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no ceiling, a computer program could be devised that uses little storage but 
has an extremely high operate time; conversely, if storage had no ceiling, a 
computer program could be devised that uses little operate time but requires 
large storage. Realistically, of course, a constraint must exist for both 
timing and storage. An optimum balance of the two must be determined con- 
sistent with other system constraints. 

The fact that the full scope of the cycle time problem was not evident until 
PST can be attributed in part to the emphasis placed on meeting the computer 
program storage constraint. Difficulty was experienced in confining ADP to 
the available storage. Although computer program storage allocation was care- 
fully reviewed at the Critical Design Reviews held for the CPCs, little 
attention or concern was expressed with regard to operate time. Some questions 
were asked but they were not meaningfully pursued. Another factor that may 
have tempered the frame of mind of those engaged in the development of ADP is 
the previous experience of SAGE. High operate times were known to occur, but 
little had been done to ascertain what adverse consequences had resulted. A 
sort of mental euphoria had set in such that the potential of a further 
timing problem was assumed to be inconsequential 

Frame time in SAGE is the equivalent to bicycle time in BUIC III. Under normal 
conditions, SAGE has a frame time of 15.7 seconds but instances of frame time 
in the range of 20 to 30 seconds have been experienced without any apparent 
effects, and are not uncommon. Concern has been expressed by personnel newly 
assigned to SAGE with regard to high frame time; however, experienced personnel 
have learned to cope with it (e.g., reduce amount of radar data processed by 
manual switch intervention). Moreover, much of the high frame time in SAGE 
has been experienced while the simulation and recording functions were 
operating, both of which increase frame time. Neither of these functions would 
be operated during a hostile attack. This analogy between SAGE frame time and 
BUIC bicycle time serves to illustrate that the relation of operate time to 
system performance may be complex and equivocal. 

3. Recommendations 

When discussing how this type of problem may be prevented or at least detected 
earlier in the future, the following should be considered: 

a. System engineering analyses and studies preceding program 
development should carefully consider system requirements 
in terms of response time desired for specified actions 
and situations, the frequency of functions, required data 
retention and allowable data loss, and similar requirements 
that influence computer program operate time. However, the 
emphasis should be on the basic requirements, not on operate 
time per se which is derived in satisfying the requirements. 
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The studies should consider alternatives and determine 
potential consequences of less than optimum performance. 

Specification of computer program requirements should 
include a description of specific load conditions that 
influence operate time for each computer program function. 
Actual values should be specified and related explicitly 
to the expected response of the CPCEI. 

The proposed program design presented at the time of 
Preliminary Design Review should include a description 
of the expected operate time of each CPC and the CPCEI 
as a whole under the varying load conditions specified. 

Personnel engaged in the management review process 
should examine and evaluate information based on test 
data regarding operate time of the CPCs during program 
development. Preliminary Qualification Tests can serve 
as the vehicle for this purpose. 

Provision should be made for the development and use of 
computer program test tools that clock the operate time 
of sets of code. Other special purpose timing tools may 
also be desired. 

Following the initial testing of the CPCs (parameter and 
assembly) typically performed by the programmers who did 
the original coding, testing should be performed by an 
independent group of programmers to offset the natural 
bias of the original programmers. 

It is evident that some problems by their nature are not 
readily detected in a simulated environment; therefore, 
it is desirable to exercise the CPCEI in a live environ- 
ment at the earliest reasonable time. 
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CHAPTER VII 
SUMMARY AND RECOMMENDATIONS 



A. INTRODUCTION 

Systems management techniques based on 375-series principles were adapted and 
applied comprehensively for the first time, in BUIC III, t9 the acquisition 
of system information processing elements. The first part of this report 
is devoted to a review of the background of those techniques in terms of Air 
Force systems management concepts and trends, and relating them to earlier 
practice at ESD in the system programs which preceded BUIC III. A second part 
of the report is devoted to describing the elements of contractor effort, 
identifying the novel management requirements, and summarizing the milestones 
associated with the requirements as they actually occurred during the course 
of the BUIC III acquisition. Finally, discussions are presented of the con- 
tractor successes and difficulties in implementing the requirements, in the 
attempt to identify and record aspects of the BUIC III experience that can be 
usefully related to future system acquisitions. 

This chapter briefly summarizes selected highlights of the findings, primarily 
in terms of comments and recommendations concerning the formally documented 
management techniques. Emphasis is placed on the formal requirements, not only 
because they were indeed the focal topic of the study, but because they provide 
the most direct vehicle by which results of experience can be brought to bear 
on future practice. At the time this report is being written, some of the 
contractual exhibits and supporting data items are in process of being revised 
to conform with recently promulgated Defense Department standards for configura- 
tion management. Similar revisions will have to be made in the related 375- 
series manuals and other documents. Based on current information, it appears 
that many of the familiar Air Force terms — e.g., names of baselines, specifi- 
cations, inspections — will be replaced by the standard DOD terminology, but 
that the existing Air Force practices may otherwise be retained essentially 
intact. Hence, it is assumed that lessons learned from the application of 
systems management techniques as discussed herein will continue to be pertinent 
following translation into the new framework of manuals and exhibits. 



B. CONFIGURATION MANAGEMENT 

Configuration management requirements in BUIC III were contractually specified 
by a special BUIC III exhibit which was equivalent in all essential respects 
to ESD Exhibit EST-1 and applicable portions of AFSCM 375-1. Basic procedures 
relating to control of changes, specification maintenance, and accounting were 



103 



similar to those which had been developed and used prior to BUIC III except for 
differences in standard forms and the fact that control had not previously been 
applied at the product configuration baseline level. In general — i.e., as 
regards configuration management aspects other than the content of specifi- 
cations — the procedures were applied to effectively control and process a 
large volume of changes during the course of the program, with relatively minor 
difficulties. Recommendations below are based on applications of EST-1 to 
other systems as well as on the BUIC III experience. 

1. Having all configuration management requirements which apply to com- 
puter programs in a self-contained document, as was true in BUIC III, is 
desirable. Although it is presently a separate document in its interim form as 
an ESD Exhibit, EST-1 is formated as change pages to AFSCM 375-1 and is more 
difficult for people concerned solely with CPCEIs to use. If and when EST-1 and 
375-1 (or their successors) are combined into a single document covering all 
types of system items, the difficulty may be increased, particularly for those 
contractors who do not have a backlog of experience in distinguishing the 
applicable principles and practice as between computer program and equipment 
items. A separate and self-contained manual for computer programs, containing 
explanations and examples in addition to the bare definitions of legal require- 
ments, would facilitate contractual application and use by both SPO and con- 
tractor personnel. 

2. The use of a common, standard format for certain forms — e.g.. 
Engineering Change Proposal — is now required for both equipment and computer 
program items. There is no advantage to this particular standardization; in 
fact, it encourages expectations of similarities which do not exist. It is 
recommended that separate forms appropriate to the purpose be devised and used, 
in the interests of improving efficiency .and avoiding misinterpretations. 

C. SPECIFICATIONS 

Requirements for the two-part CPCEI specification imposed in BUIC III were 
essentially those which are currently described in EST-1 and associated Forms 
9 (now DD 1664s), and for which supplementary experience is also available from 
their use in other programs. Based on these experiences certain recommended 
lines of improvement are suggested in Chapter VI preceding, and below. However, 
to maintain perspective, the problems identified should be construed as 
indicating only that there are specific areas needing further study and refine- 
ment, not that basic changes are being proposed. It is believed that the two- 
part specification has been demonstrated to be the key element in developing 
an effective framework for managing computer program acquisition, and that the 
existing structure of requirements is superior to various proposed alternatives. 

The types and levels of information to be supplied in the Part I specification 
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are generally appropriate and necessary in order for the Part I to accomplish 
its various functions, namely: of governing the computer program design, 
development, and testing; controlling interfaces of the CPCEI with other items; 
and serving as the primary baseline for CPCEI configuration management. The 
Part II meets the important needs of providing comprehensive documentation 
which can be used by computer programmers for diagnosing malfunctions and 
designing changes, as well as providing a precise technical definition of the 
CPCEI for configuration management at the product baseline level. However, 
problems needing further study and clarification include: 

1. The degree to which considerations of computer program design are in- 
corporated into the content and structure of the Part I specification should 
be determined. Questions exist, on the one hand, whether mathematical for- 
mulas and equations representing design solutions, which may conflict with 
requirements expressed purely in performance terms, should be called for in 
the Part I except where their firm intent is to establish design constraints. 
On the other hand, the question has also been raised whether the overall com- 
puter program design structure should not be anticipated, verified, and est- 
ablished in a corresponding structure of Part I functional elements, to provide 
an improved basis both for managing the computer program development and for 
testing. 

2. The scope of intended coverage under Data Base Requirements (3.1.3) 
in the Part I specification should be determined, and a number of terms should 
be defined. Questions needing resolution relate to data which are input prior 
to computer program operation vs. variable values input during the course of 
operation, computational constants vs. data to be processed, input/output vs. 
intermediate data, and others. The term "data base" is subject to a variety 
of interpretations with respect to these considerations. Also, a number of 
terms, e.g., "adaptation," "parameter," etc., need to be defined and supported 
by explanations of their significance in terms of functions served by the speci- 
fication. 

3. The level of descriptive detail contained in the Part II specifications 
should be examined. Experience has indicated that the Form 9 (DD 1664) for 

the Part II specification is comprehensive and technically sound. However, as 
is true of Forms 9 in general, its application should be carefully tailored 
to the given CPCEI, taking into account the intended uses and maintenance of 
the document following its acceptance. The value of detailed prose and flow 
chart materials at the CPC level, in particular, which are expensive to pro- 
duce and maintain, should be examined critically, and the required detail 
reduced to a minimum level which is consistent with the needs of the given 
application. 

4. Implied functions of the Part II specification as a contractual require- 
ments instrument should be clarified. The Part II computer program specifica- 
tion is recognized in EST-1 to be an "as-built" technical description of the 
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CPCEI configuration. However, the wording and content of certain sections in 
the Form 9 are not fully consistent with that role, e.g., as contrasted with 
functions which an equipment Part II specification would normally serve in 
governing production, acceptance, and delivery, following the first article. 
Recommended changes which might help to avoid confusion and misinterpretations 
include: under Section 1, "SCOPE", eliminate emphasis on units subsequent to 
the FACI'd article and on CPCEI serial numbers; as a title for Section 3, 
substitute "TECHNICAL DESCRIPTION" for "REQUIREMENTS"; eliminate Sections A and 
5 completely, as types of requirements which should be contractually specified 
independently of the Part II CPCEI specification, prior to delivery of the 
first article. 



D. DESIGN REVIEWS 

A Preliminary Design Review (PDR) was held for each of the three major BUIC III 
CPCEIs, essentially in accordance with the guidelines set forth in ESD Exhibits 
EST-1 and EST-3. These were conducted successfully and were regarded as being 
useful and productive sessions by representatives of both the SPO and contractor. 

Critical Design Reviews (CDRs) were conducted incrementally in combination with 
Preliminary Qualification Tests, at the level of completed design for the com- 
ponents undergoing review. These provided opportunities for review and comment 
by representatives of the SPO, and were considered useful. Although some of 
the discussions are reported to have resulted in minor changes, the CDRs were 
not associated with formal actions. 

As a general matter, questions continue to be raised regarding objectives, in 
terms of formal actions, which the computer program CDR is intended to accomplish. 
Established objectives for equipment — e.g., release of design fot production 
and as a basis for support items — do not apply. Available guidance in EST-1 
and EST-3 indicates that the only analogous purpose for CPCEIs is to identify 
documentation which will be "released for coding and testing", a purpose which 
can only exist when CDRs are held at the flow chart level. At that stage the 
development is only partially complete and no testing can have been accomplished; 
implications relating to formal approval/disapproval, and applicability of 
configuration controls, have become matters of debate. The problem deserves 
further attention. In the interim, it is suggested that in the case of com- 
puter programs, CDRs should be regarded frankly as monitoring events only, at 
a level between PDR and FACT, having the primary purpose of providing visi- 
bility of the development. A change in the name would be a helpful step towards 
alleviating the prevalent confusion with accepted practices associated with 
CDRs for equipment items. 
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E. CATEGORY I TESTING 

Requirements for Category I testing of CPCEIs encompassed: identification of 
test requirements in the Part I CPCEI specifications; preparation of Category 
I Test Plans; preparation of PQT/FQT Test Procedures and Reports; and conduct 
of the formal test sessions. As a whole, these represented elements of con- 
tractor effort which were novel in BUIC III as compared with practice in 
earlier systems. Difficulties were encountered in carrying out the test pro- 
gram which are attributed to absence of previous contractor experience with 
the concepts and procedures, as well as to certain ambiguities in the require- 
ments as they were formulated at the time of contractual application. 

The principal difficulties were associated with PQTs , which were carried out in 
a total of 57 testing sessions for two of the CPCEIs (ADP and SEP) and were 
designed to provide comprehensive verification of performance relating to 
detailed characteristics specified in the Part I specifications. The PQTs 
proved to be costly in time and effort to conduct and were not particularly 
satisfactory in meeting their objectives. 

In the case of FQTs, the principal objective was to demonstrate compliance 
with major performance requirements of the computer programs during integrated 
system operation. Although some problems were encountered with documentation 
and test conduct, particularly during early attempts, the FQTs were compara- 
tively successful. In contrast with the PQTs, they were readily recognized by 
the contractor personnel involved as being formal tests which serve useful 
and necessary purposes, as well as being feasible to accomplish. 

Comments and recommendations based on discussions contained in the preceding 
chapter are summarized briefly below. 

1. A number of factors can be identified as contributing to the severe 
problems experienced in carrying out the BUIC III PQTs. In some part these 
were matters of contractor management and technical practices which had not 
been geared to meeting the complex requirements of formal Category I testing, 
and which did not receive the vigorous attention they needed until late in the 
program. However, the authors believe that the initial approach which the SPO 
and contractor devised jointly at an early point was basically unsound. The 
PQTs planned were far too many and attempted to be too comprehensive in their 
coverage; they should have been regarded and planned as interim demonstrations 
of contractor progress towards accomplishing the development, not as means of 
qualification. 

2. To be realistic, qualification requirements must be tailored to 
individual cases. The degree to which comprehensive formal verification is 
feasible, and justifies its cost, should be recognized as a variable which 
depends on such considerations as size and complexity of the CPCEI, frequency 
and extent of changes, and criticality of perfect performance in initial 
operational use. 
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3. The contractor should be required to demonstrate internal management 
procedures which insure that Part I specification requirements, including 
approved changes, are incorporated into the computer program design and tested 
routinely during CPT&E. Where conditions permit, documented CPT&E procedures 
and results should be utilized as a source of data for qualifying detailed 
performance/design characteristics of the computer program components. 

4. Where feasible, it is desirable to conduct FQTs using the integrated 
CPCEI in the system environment. To the degree that adequate simulation is 
available, FQTs or partial FQTs should be performed at the contractor's 
facility. 



F. CONCLUSION 

As a major summary result, this study has indicated that the principles and 
content of the management requirements are not only sound but represent 
necessary and substantial improvements over previous practice. Their appli- 
cation in BUIC III occasioned problems, principally in the areas of documenta- 
tion and testing, which are discussed at length in the preceding chapter. In 
general, however, the problems can be related most directly to the fact that 
the procedures were novel, rather than unsound, and the BUIC III personnel 
were inexperienced with their use. The original intent was to initiate the 
adoption of standards which would enable computer programming to be managed 
in a uniform manner, consistently with the management of equipment and other 
elements in a system program. The matters of uniformity and continuity in 
documentation and procedures are the key elements which make it possible for 
both SPO and contractor to benefit from experience, from system to system. In 
the authors' opinion, these advantages have become more evident as the pro- 
cedures have become better understood and established, particularly through 
continued use in other system programs. 

By the same token, BUIC III clearly demonstrated that the initial acquisition 
of contractor capability to implement the requirements adequately can be 
difficult and time-consuming. In part, some of the problems can be ascribed 
to the fact that the total 375-series framework was a novelty to the con- 
tractor at the time BUIC III began, and many of its implications for Internal 
management were not appreciated until late in the program. There is evidence 
that difficulties could have been avoided by better initial guidance, e.g., 
in interpreting the formal test requirements, and by systematic training of 
personnel in relevant documentation and procedures. Misinterpretations and 
absence of common understanding of objectives and requirements were frequent, 
both internally among contractor personnel and in relations with the SPO. 
In general, as is indicated in comments and recommendations made above, pro- 
blems can be traced to a variety of sources. However, it is believed that 
training of personnel, based on adequate uniform guidance, will continue to 
be a fruitful area for increased future attention. 
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APPENDIX A 
GLOSSARY OF ABBREVIATIONS 



AC&W 

ADC 

ADP 

AFL 

AFLC 

AFR 

AFS 

AFSC 

AFSC 

AFS CM 

AGE 

AI&C 

APASTO 

ARDC 

AS PR 

ATC 

BARS 

BCDP 

BEF 

BEPS 

BMF 

BSAO 

BSD 

BUDR 

BUIC 

CCB 

CCBD 

CCDSO 

CDP 

CDR 

CDRL 

CEI 

CMO 

CC 

CP 

CPC 

CPCEI 

CPT&E 

CR 

CR 



Aircraft Control and Warning 

Aerospace Defense Command (formerly Air Defense Command) 

Air Defense (Computer) Program 

Air Force Letter 

Air Force Logistics Command 

Air Force Regulation 

Air Force Specialty 

Air Force Specialty Code 

Air Force Systems Command 

Air Force Systems Command Manual 

Aerospace Ground Equipment 

Adaptation, Installation, and Checkout 

Air Defense Computer Program and System Training Office 

Air Research and Development Command (predecessor of AFSC) 

Armed Services Procurement Regulation 

Air Training Command 

BUIC Analysis and Reduction System 

Backup Confidence Diagnostic Computer Program 

BUIC Evaluation Facility 

BUIC Exercise Preparation System 

BUIC Management File 

Burroughs Site Activation Office 

Ballistic Systems Division 

BUIC Data Reduction 

Backup Interceptor Control 

Configuration Control Board 

Configuration Change Board Directive 

ADC Command and Control Defense Systems Office 

Contract Definition Phase 

Critical Design Review 

Contract Data Requirements List 

Contract End Item 

Configuration Management Office 

Control Center 

Change Proposal 

Computer Program Component 

Computer Program Contract End Item 

Computer Program Test and Evaluation 

Change Report (In BUIC III, reporting a Class II change) 

Change Recommendation (In BUIC II, a recommended change to any 

system element, prepared by any participating agency) 



109 



DC - Direction Center 

DOD - Department of Defense 

ECP - Engineering Change Proposal 

ERP - Error Recovery Program 

ESD - Electronic Systems Division 

ESS - Experimental SAGE Sector 

FACT - First Article Configuration Inspection 

FQT - Formal Qualification Test 

FSPS - Field Site Production System 

GEEIA - Ground Electronics Engineering and Installation Agency 

GFP - Government-Furnished Property 

GSE/TDC - General Systems Engineering/Technical Direction Contractor 

HE - Human Engineering 

lAC - Integrating and Assembly Contractor 

LET - Live Environment Testing 

LLO - Lexington Liaison Office 

LS - Life Support 

MSTP - Manual System Training Program 

NCC - NORAD Control Center 

NORAD - North American Air Defense Command 

NRD - National Range Division 

OR - Operational Requirement 

PAGE - Primary Air Ground Environment 

PCB - Product Configuration Baseline 

PDR - Preliminary Design Review 

PECP - Preliminary Engineering Change Proposal 

PED - Personnel-Equipment Data 

PERT - Program Evaluation and Review Technique 

PQT - Preliminary Qualification Test 

PS - Personnel Subsystem 

PSTE - Personnel Subsystem Test and Evaluation 

PSPP - Proposed System Package Plan 

PST - Program System Test 

PTDP - Preliminary Technical Development Plan 

QQPRI - Qualitative and Quantitative Personnel Requirements Information 
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RDT&E 
RFP 



Research, Development, Test and Evaluation 
Request for Proposal 



SAC 

SAC 

SAGE 

SAWG 

SCL 

sen 

SATAF 

SDC 

SDR 

SEP 

SETE 

SOW 

SPD 

SPO 

SPP 

SSTP 



Strategic Air Command 

Site Activation Contractor 

Semi-Automatic Ground Environment (System) 

System Acquisition Working Group 

Specification Change Log 

Specification Change Notice 

Site Activation Task Force 

System Development Corporation 

System Design Review 

System Exercise (Computer) Program 

System Exercising for Training and Evaluation 

Statement of Work 

System Program Director 

System Program Office 

System Package Program 

SAGE System Training Program 



TAC 

TC 

TED 

TEPI 

TM 

TP 



Tactical Air Command 

Training Concept 

Training Equipment Development 

Training Equipment Planning Information 

Technical Memorandum 

Training Plan 



UCP 
USAF 
USCN 
USP 



Utility Computer Program 

United States Air Force 

Uniform Specification Change Notice 

Uniform Specification Program 



VDE 



Variable Display Equipment 
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